Skip to content

Null generated environment settings #418

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
warsaw opened this issue Dec 5, 2016 · 8 comments
Closed

Null generated environment settings #418

warsaw opened this issue Dec 5, 2016 · 8 comments

Comments

@warsaw
Copy link

warsaw commented Dec 5, 2016

I could have sworn I reported this before, but I can't find it. Maybe I only mentioned it on the mailing list, or described it differently on the old Bitbucket tracker. Anyway...

Let's say I want the following environments:

  • Python 3.4, 3.5, 3.6
  • Python 3.4, 3.5, 3.6 with coverage
  • Python 3.4, 3.5, 3.6 with diffcov

Currently I have to specify them like this:

[tox]
envlist = {py34,py35,py36}-{cov,nocov,diffcov},qa,docs

[testenv]
commands =
    nocov: python -m nose2 -v {posargs}
    {cov,diffcov}: python -m coverage run {[coverage]rc} -m nose2 -v
    {cov,diffcov}: python -m coverage combine {[coverage]rc}
    cov: python -m coverage html {[coverage]rc}
    cov: python -m coverage report -m {[coverage]rc} --fail-under=100
    diffcov: python -m coverage xml {[coverage]rc}
    diffcov: diff-cover coverage.xml --html-report diffcov.html
    diffcov: diff-cover coverage.xml --fail-under=100

And I have to run a "no coverage Python 3.5" test like this: tox -e py35-nocov

But what I'd really like to do is to say tox -e py35 for the "no coverage" case. However I can't do this:

[tox]
envlist = {py34,py35,py36}{,-cov,-diffcov},qa,docs
recreate = True
skip_missing_interpreters = True

[testenv]
commands =
    python -m nose2 -v {posargs}
    {cov,diffcov}: python -m coverage run {[coverage]rc} -m nose2 -v
    {cov,diffcov}: python -m coverage combine {[coverage]rc}
    cov: python -m coverage html {[coverage]rc}
    cov: python -m coverage report -m {[coverage]rc} --fail-under=100
    diffcov: python -m coverage xml {[coverage]rc}
    diffcov: diff-cover coverage.xml --html-report diffcov.html
    diffcov: diff-cover coverage.xml --fail-under=100

because the first command, i.e. python -m nose2 -v {posargs} will run for every test environment, which is definitely not what I want. I need a way to run commands when they don't match a generated environment. Perhaps something like this would do the trick?

[testenv]
commands =
    : python -m nose2 -v {posargs}

It's possible of course that I'm missing something obvious.

@speedyleion
Copy link

speedyleion commented Jan 16, 2017

Note: This suggestion is incorrect. Keeping it here for the conversation history.

I had the same desire and I found a way, thought it maybe leveraging an unintentional feature...

[tox]
envlist = {py27, py35, py36}{,-cov}

[testenv]
commands =
    # The magic is the empty {} after the python version list
    {py27,py35,py36}{}: python -m nose2 -v {posargs}
    cov: python -m coverage html {[coverage]rc}

This is (unfortunatly) on windows using tox 2.4.1

@warsaw
Copy link
Author

warsaw commented Jan 16, 2017

That would actually be cool, but it doesn't work for me. Note that I'm using tox 2.5.0 on Linux.

[tox]
envlist = {py34,py35,py36}{,-cov,-diffcov}{,-mysql,-pg},qa

[testenv]
commands =
    {py34,py35,py36}{}: python -m nose2 -v {posargs}
    cov,diffcov: python -m coverage run {[coverage]rc} -m nose2 {posargs}
    cov,diffcov: python -m coverage combine {[coverage]rc}
    cov: python -m coverage html {[coverage]rc}
    cov: python -m coverage report -m {[coverage]rc} --fail-under=93
    diffcov: python -m coverage xml {[coverage]rc}
    diffcov: diff-cover coverage.xml --html-report diffcov.html
    diffcov: diff-cover coverage.xml --fail-under=100

tox -l shows py35 as a testenv, but using tox --showconfig gives me:

[testenv:py35]
  envdir          = /home/barry/projects/mailman/core/.tox/py35
  setenv          = <tox.config.SetenvDict object at 0x7f7eae0a4a20>
  basepython      = python3.5
  envtmpdir       = /home/barry/projects/mailman/core/.tox/py35/tmp
  envlogdir       = /home/barry/projects/mailman/core/.tox/py35/log
  downloadcache   = None
  changedir       = /home/barry/projects/mailman/core
  args_are_paths  = True
  skip_install    = False
  ignore_errors   = False
  recreate        = False
  passenv         = {'TMPDIR', 'PATH', 'LANG', 'PIP_INDEX_URL', 'PYTHONPATH', 'PYTHONSTARTUP', 'LD_LIBRARY_PATH'}
  whitelist_externals = []
  platform        = .*
  sitepackages    = False
  pip_pre         = False
  usedevelop      = True
  install_command = ['pip', 'install', '{opts}', '{packages}']
  list_dependencies_command = ['pip', 'freeze']
  deps            = [flufl.testing, nose2]
  commands        = []
  ignore_outcome  = False
  extras          = []

Note that commands is empty. commands are correct for the non-"null" environments.

@hpk42 The empty {} would be acceptable syntax for me. Maybe the feature could be resurrected and documented?

@speedyleion
Copy link

oh you're right.
i fail hard on that.
I was just seeing success at the bottom but it doesn't actually run the command sorry for the misdirection there :(.

@speedyleion
Copy link

So a workaround that does work...

envlist = py{27,35,36}{,-cov}

[testenv]
commands =
    cov: python -m coverage run {[coverage]rc} -m nose2  -v {posargs}
    cov: python -m coverage combine {[coverage]rc}
    cov: python -m coverage html {[coverage]rc}
    cov: python -m coverage report -m {[coverage]rc} --fail-under=90
   
[testenv:py27]
commands =
    nose2 -v {posargs}

[testenv:py35]
commands =
    nose2 -v {posargs}

[testenv:py36]
commands =
    nose2 -v {posargs} 

Basically explicitly create sections for the "normal" runs and specify the command there in each run. This could be made more maintainable by utilizing section substitutions, https://tox.readthedocs.io/en/latest/config.html#substitution-for-values-from-other-sections

I did a dump of tox --showconfig before and after, and the only differences were the populated commands fields and the object instance addresses.

@warsaw
Copy link
Author

warsaw commented Jan 17, 2017

@speedyleion Thanks, that does work, but it's less convenient the more factors are involved. E.g. I have three dimensions of factors, the Python version (3.4, 3.5, and 3.6), the coverage type (no-coverage, coverage, diff_cover), and the database type (sqlite, mysql, pg).

I still kind of like the empty {} idea. :)

@speedyleion
Copy link

speedyleion commented Jan 17, 2017

I agree, it would be nice to have a way to specify project defaults, without forcing a default factor.
I just happened to also be looking at shrinking my default environment names on the command line.

Talking with a colleague we were thinking another idea is to put them in the [tox] group.

[tox]
# declaring here creates a default value for all environments
commands =
    nose2 -v {posargs}

[testenv]
# Using a factor will override the default for that factor
commands =
    cov: python -m coverage run {[coverage]rc} -m nose2  -v {posargs}
    cov: python -m coverage combine {[coverage]rc}

[testenv:py27]
# Doing it inside of an environment group would override the default for the environment
commands =
    python -m nose2 -v {posargs}

@warsaw
Copy link
Author

warsaw commented Jan 25, 2017

I was thinking about this last night and I realized that a solution to #238 could also solve this problem.

If you combine the idea of groups and aliases, then it might not matter that you don't have null environments because you can hide the less convenient py35-cov under an alias. E.g. imagine:

[tox]
envlist = {py35,py36}-{nocov,cov,diffcov},qa
aliases =
    default: {envlist}
    py35: py35-nocov
    py36: py36-nocov
    coverage: {py35,py36}-cov
    nocov: {py35,py36}-nocov

Now I could do the following:

$ tox -g py35  # Runs the py35-nocov commands
$ tox -g coverage # Runs all coverage tests
$ tox -g nocov # Runs all the standard tests

This gives me my succinct cli invocation, which is the real motivation behind the request for null environments. It means that I don't need null environments in my envlist; I just invoke the tests I care about through the -g option (i.e. run a group, or alias, environment).

@obestwalter
Copy link
Member

closing in favour of #238

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants