Null generated environment settings

[warsaw]

I could have sworn I reported this before, but I can't find it. Maybe I only mentioned it on the mailing list, or described it differently on the old Bitbucket tracker. Anyway...

Let's say I want the following environments:

• Python 3.4, 3.5, 3.6
• Python 3.4, 3.5, 3.6 with coverage
• Python 3.4, 3.5, 3.6 with diffcov

Currently I have to specify them like this:

[tox]
envlist = {py34,py35,py36}-{cov,nocov,diffcov},qa,docs

[testenv]
commands =
    nocov: python -m nose2 -v {posargs}
    {cov,diffcov}: python -m coverage run {[coverage]rc} -m nose2 -v
    {cov,diffcov}: python -m coverage combine {[coverage]rc}
    cov: python -m coverage html {[coverage]rc}
    cov: python -m coverage report -m {[coverage]rc} --fail-under=100
    diffcov: python -m coverage xml {[coverage]rc}
    diffcov: diff-cover coverage.xml --html-report diffcov.html
    diffcov: diff-cover coverage.xml --fail-under=100

And I have to run a "no coverage Python 3.5" test like this: tox -e py35-nocov

But what I'd really like to do is to say tox -e py35 for the "no coverage" case. However I can't do this:

[tox]
envlist = {py34,py35,py36}{,-cov,-diffcov},qa,docs
recreate = True
skip_missing_interpreters = True

[testenv]
commands =
    python -m nose2 -v {posargs}
    {cov,diffcov}: python -m coverage run {[coverage]rc} -m nose2 -v
    {cov,diffcov}: python -m coverage combine {[coverage]rc}
    cov: python -m coverage html {[coverage]rc}
    cov: python -m coverage report -m {[coverage]rc} --fail-under=100
    diffcov: python -m coverage xml {[coverage]rc}
    diffcov: diff-cover coverage.xml --html-report diffcov.html
    diffcov: diff-cover coverage.xml --fail-under=100

because the first command, i.e. python -m nose2 -v {posargs} will run for every test environment, which is definitely not what I want. I need a way to run commands when they don't match a generated environment. Perhaps something like this would do the trick?

[testenv]
commands =
    : python -m nose2 -v {posargs}

It's possible of course that I'm missing something obvious.

[speedyleion]

Note: This suggestion is incorrect. Keeping it here for the conversation history.

I had the same desire and I found a way, thought it maybe leveraging an unintentional feature...

[tox]
envlist = {py27, py35, py36}{,-cov}

[testenv]
commands =
    # The magic is the empty {} after the python version list
    {py27,py35,py36}{}: python -m nose2 -v {posargs}
    cov: python -m coverage html {[coverage]rc}

This is (unfortunatly) on windows using tox 2.4.1

[warsaw]

That would actually be cool, but it doesn't work for me. Note that I'm using tox 2.5.0 on Linux.

[tox]
envlist = {py34,py35,py36}{,-cov,-diffcov}{,-mysql,-pg},qa

[testenv]
commands =
    {py34,py35,py36}{}: python -m nose2 -v {posargs}
    cov,diffcov: python -m coverage run {[coverage]rc} -m nose2 {posargs}
    cov,diffcov: python -m coverage combine {[coverage]rc}
    cov: python -m coverage html {[coverage]rc}
    cov: python -m coverage report -m {[coverage]rc} --fail-under=93
    diffcov: python -m coverage xml {[coverage]rc}
    diffcov: diff-cover coverage.xml --html-report diffcov.html
    diffcov: diff-cover coverage.xml --fail-under=100

tox -l shows py35 as a testenv, but using tox --showconfig gives me:

[testenv:py35]
  envdir          = /home/barry/projects/mailman/core/.tox/py35
  setenv          = <tox.config.SetenvDict object at 0x7f7eae0a4a20>
  basepython      = python3.5
  envtmpdir       = /home/barry/projects/mailman/core/.tox/py35/tmp
  envlogdir       = /home/barry/projects/mailman/core/.tox/py35/log
  downloadcache   = None
  changedir       = /home/barry/projects/mailman/core
  args_are_paths  = True
  skip_install    = False
  ignore_errors   = False
  recreate        = False
  passenv         = {'TMPDIR', 'PATH', 'LANG', 'PIP_INDEX_URL', 'PYTHONPATH', 'PYTHONSTARTUP', 'LD_LIBRARY_PATH'}
  whitelist_externals = []
  platform        = .*
  sitepackages    = False
  pip_pre         = False
  usedevelop      = True
  install_command = ['pip', 'install', '{opts}', '{packages}']
  list_dependencies_command = ['pip', 'freeze']
  deps            = [flufl.testing, nose2]
  commands        = []
  ignore_outcome  = False
  extras          = []

Note that commands is empty. commands are correct for the non-"null" environments.

@hpk42 The empty {} would be acceptable syntax for me. Maybe the feature could be resurrected and documented?

[speedyleion]

oh you're right.
i fail hard on that.
I was just seeing success at the bottom but it doesn't actually run the command sorry for the misdirection there :(.

[speedyleion]

So a workaround that does work...

envlist = py{27,35,36}{,-cov}

[testenv]
commands =
    cov: python -m coverage run {[coverage]rc} -m nose2  -v {posargs}
    cov: python -m coverage combine {[coverage]rc}
    cov: python -m coverage html {[coverage]rc}
    cov: python -m coverage report -m {[coverage]rc} --fail-under=90
   
[testenv:py27]
commands =
    nose2 -v {posargs}

[testenv:py35]
commands =
    nose2 -v {posargs}

[testenv:py36]
commands =
    nose2 -v {posargs} 

Basically explicitly create sections for the "normal" runs and specify the command there in each run. This could be made more maintainable by utilizing section substitutions, https://tox.readthedocs.io/en/latest/config.html#substitution-for-values-from-other-sections

I did a dump of tox --showconfig before and after, and the only differences were the populated commands fields and the object instance addresses.

[warsaw]

@speedyleion Thanks, that does work, but it's less convenient the more factors are involved. E.g. I have three dimensions of factors, the Python version (3.4, 3.5, and 3.6), the coverage type (no-coverage, coverage, diff_cover), and the database type (sqlite, mysql, pg).

I still kind of like the empty {} idea. :)

[speedyleion]

I agree, it would be nice to have a way to specify project defaults, without forcing a default factor.
I just happened to also be looking at shrinking my default environment names on the command line.

Talking with a colleague we were thinking another idea is to put them in the [tox] group.

[tox]
# declaring here creates a default value for all environments
commands =
    nose2 -v {posargs}

[testenv]
# Using a factor will override the default for that factor
commands =
    cov: python -m coverage run {[coverage]rc} -m nose2  -v {posargs}
    cov: python -m coverage combine {[coverage]rc}

[testenv:py27]
# Doing it inside of an environment group would override the default for the environment
commands =
    python -m nose2 -v {posargs}

[warsaw]

I was thinking about this last night and I realized that a solution to #238 could also solve this problem.

If you combine the idea of groups and aliases, then it might not matter that you don't have null environments because you can hide the less convenient py35-cov under an alias. E.g. imagine:

[tox]
envlist = {py35,py36}-{nocov,cov,diffcov},qa
aliases =
    default: {envlist}
    py35: py35-nocov
    py36: py36-nocov
    coverage: {py35,py36}-cov
    nocov: {py35,py36}-nocov

Now I could do the following:

$ tox -g py35  # Runs the py35-nocov commands
$ tox -g coverage # Runs all coverage tests
$ tox -g nocov # Runs all the standard tests

This gives me my succinct cli invocation, which is the real motivation behind the request for null environments. It means that I don't need null environments in my envlist; I just invoke the tests I care about through the -g option (i.e. run a group, or alias, environment).

[obestwalter]

closing in favour of #238