tox doesn't strip line coments from dependencies

[pytoxbot]

• Bitbucket: https://bitbucket.org/hpk42/tox/issue/332
• Originally reported by: @The-Compiler
• Originally created at: 2016-05-13T11:24:07.337

With an up-to-date virtualenv (which downloads an up-to-date pip), since pip 8.1.2, tox breaks when there's a tox.ini like this:

[testenv]
deps=
  pytest # comment

output:

GLOB sdist-make: /home/florian/tmp/setup.py
python create: /home/florian/tmp/.tox/python
python installdeps: pytest # comment
ERROR: invocation failed (exit code 1), logfile: /home/florian/tmp/.tox/python/log/python-1.log
ERROR: actionid: python
msg: getenv
cmdargs: [local('/home/florian/tmp/.tox/python/bin/pip'), 'install', 'pytest # comment']
env: {...}

Invalid requirement: 'pytest # comment'
Traceback (most recent call last):
  File "/home/florian/tmp/.tox/python/lib/python3.4/site-packages/pip/_vendor/packaging/requirements.py", line 92, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/home/florian/tmp/.tox/python/lib/python3.4/site-packages/pip/_vendor/pyparsing.py", line 1172, in parseString
    raise exc
  File "/home/florian/tmp/.tox/python/lib/python3.4/site-packages/pip/_vendor/pyparsing.py", line 1162, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/home/florian/tmp/.tox/python/lib/python3.4/site-packages/pip/_vendor/pyparsing.py", line 1028, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/florian/tmp/.tox/python/lib/python3.4/site-packages/pip/_vendor/pyparsing.py", line 2462, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/home/florian/tmp/.tox/python/lib/python3.4/site-packages/pip/_vendor/pyparsing.py", line 1032, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/florian/tmp/.tox/python/lib/python3.4/site-packages/pip/_vendor/pyparsing.py", line 2265, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected stringEnd (at char 7), (line:1, col:8)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/florian/tmp/.tox/python/lib/python3.4/site-packages/pip/req/req_install.py", line 78, in __init__
    req = Requirement(req)
  File "/home/florian/tmp/.tox/python/lib/python3.4/site-packages/pip/_vendor/packaging/requirements.py", line 96, in __init__
    requirement_string[e.loc:e.loc + 8]))
pip._vendor.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'# commen'"


ERROR: could not install deps [pytest # comment]; v = InvocationError('/home/florian/tmp/.tox/python/bin/pip install pytest # comment (see /home/florian/tmp/.tox/python/log/python-1.log)', 1)

This is because pip doesn't handle comments as part of a commandline argument since that version anymore: pypa/pip#3667

They replied in that issue that it isn't really intended that it worked before, and I tend to agree.

Marking as critical as this completely breaks tox unless you set VIRTUALENV_DOWNLOAD=no and never upgrade virtualenv.

I think tox should either strip those comments (no idea how tricky that is as pip URLs can include hashes as well), or write temporary requirement.txt files and direct pip to them.

[pytoxbot]

Original comment by @jayvdb

i.e. pypa/pip#3667 was closed as wontfix; it is now up to tox to fix this.

[pytoxbot]

Original comment by @The-Compiler

FWIW the underlying pip issue has been closed now and the behaviour won't be changed back on their side.

[pytoxbot]

Original comment by @jayvdb

issue title spelling fix... coments -> comments

[pytoxbot]

Original comment by @jayvdb

Note this the workaround doesnt work on pypy3-dev (5.2) as it requires virtualenv>15.0.2

[pytoxbot]

Original comment by kalikaneko

This PR fixes the problem https://bitbucket.org/hpk42/tox/pull-requests/202/strip-any-comment-from-the-dependencies/

[pytoxbot]

Original comment by @RonnyPfannschmidt

as far as i understand, pip urls only contain hashes as part of scm metadata, those dont have whitespace right before
as such i think we can skip anything after r'[ \t]#'

[obestwalter]

Hi @kalikeneko,

interested in whipping you PR from bitbucket into shape and re open here?

[ghost]

@obestwalter I'll take this

[obestwalter]

Hi @sifuraz sorry for dropping the ball there :( I had simply forgotten about this #50