"pipenv install --system" breaks when installing from requirements.txt

[nottrobin]

$ pipenv install --system
requirements.txt found, instead of Pipfile! Converting…
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 11, in <module>
    load_entry_point('pipenv==11.1.3', 'console_scripts', 'pipenv')()
  File "/usr/local/lib/python3.5/dist-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.5/dist-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.5/dist-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.5/dist-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/pipenv/cli.py", line 197, in install
    selective_upgrade=selective_upgrade
  File "/usr/local/lib/python3.5/dist-packages/pipenv/core.py", line 1731, in do_install
    ensure_project(three=three, python=python, system=system, warn=True, deploy=deploy, skip_requirements=skip_requirements)
  File "/usr/local/lib/python3.5/dist-packages/pipenv/core.py", line 652, in ensure_project
    ensure_pipfile(validate=validate, skip_requirements=skip_requirements)
  File "/usr/local/lib/python3.5/dist-packages/pipenv/core.py", line 319, in ensure_pipfile
    project.create_pipfile(python=python)
  File "/usr/local/lib/python3.5/dist-packages/pipenv/project.py", line 439, in create_pipfile
    data[u'requires'] = {'python_version': python_version(self.which('python'))[:len('2.7')]}
TypeError: 'NoneType' object is not subscriptable

From what I can tell this is a bug in either the which function or the use of the which function. I'm guessing that the allow_global option isn't set when it should be.

[techalchemy]

@nottrobin Pipenv really can't be used like this. Using --system disables the virtualenv which is necessary for isolated resolution. The dependency resolver isn't designed to do resolution against system packages unless done inside a virtualenv. --system should only be used with a lockfile, we simply haven't improved error messaging around this. To build such a lockfile you might want to do pipenv install --site-packages -r <your_requirements_file> -- this will build an environment using your requirement file and resolve against the system site packages in that isolated environment, producing a Pipfile and a Pipfile.lock.

You can then pipenv --rm to delete the virtualenv if you like, and use the new files to pipenv install --system

[nottrobin]

@techalchemy From a zoomed-out perspective, I can't see any reason why the parsing of requirements.txt and creation of Pipfile and Pipefile.lock should be at all related to the environment management.

In my opinion the great thing about Pipenv is its improved version management using Pipfile & Pipfile.lock rather than the virtualenv encapsulation. The main complaints about Pipenv that I've heard are because of people's misconceptions that it is only about environment management, and they want to keep their existing environment management solutions, but still use Pipfile.

This is sort of the case in my use-case. In my team, we run all our apps and their dependencies through a Docker image, so those dependencies needn't pollute the dev's system nor be effected by the existing state of the dev's system. In the context of this docker image, it doesn't make much sense to ask Pipenv to also create its own virtualenv inside the Docker image, since the Docker image is quite enough encapsulation (and in any case is running commands as root). This is why we use Pipenv --system is every case.

It would be really nice to be able to take advantage of Pipenv's neat automatic behaviour with regard to requirements.txt files within our workflow. And I imagine that there are plenty of other contexts where people only want to use Pipenv with system python and would also appreciate it.

[techalchemy]

@nottrobin just because you can’t see why it’s related to environment management doesn’t mean it isn’t. Dependency resolution demands an isolated environment. If you can build a revolver that doesn’t require this we would be happy to evaluate it.

If you are never going to install anything anywhere besides a docker container and you insist that that is the only place dependency resolution can ever happen and your developers can’t or won’t use a virtualenv to make a lockfile, then you should just use pip directly as it will be much faster and accomplish the same thing.

[nottrobin]

Dependency resolution demands an isolated environment

I'm saying that it always used to be up to the python dev to understand which environment they were using, and it doesn't seem unreasonable to still give them the option to still have that control. In my case I know that the system is the environment I care about.

If you are never going to install anything anywhere besides a docker container and you insist that that is the only place dependency resolution can ever happen and your developers can’t or won’t use a virtualenv to make a lockfile, then you should just use pip directly as it will be much faster and accomplish the same thing.

As I say, the really valuable thing about pipenv is the Pipfile format with all its lockfile protections. You can't achieve this with pip alone. Are you saying that if we don't want to manage virtualenvs with pipenv then we should abandon the Pipfile standard as well? I thought that was meant to be the future of package management in python, so that would be a shame.

[techalchemy]

As I explained, generating a lockfile requires isolation. That isn’t a decision we made because we want to be jerks. If you think you can improve upon it, contributions are welcome. Currently you can’t build a lockfile without a virtualenv. Period.

[nottrobin]

As I explained, generating a lockfile requires isolation. That isn’t a decision we made because we want to be jerks. If you think you can improve upon it, contributions are welcome. Currently you can’t build a lockfile without a virtualenv. Period.

Ah. Sorry I didn't quite understand that technical limitation from your comment before. I see now. Yes maybe one day when I've got some free time I might delve into the way the lockfile generation works and see if I can make any suggestions. For now, suffice to say it would be nice for it to work with system python, but I understand if this is not possible.

My solution for now is to simply instruct people in our team who want to upgrade projects that still use requirements.txt to locally run:

[sudo] pip install pipenv
pipenv install
pipenv --rm
[sudo] pip uninstall pipenv
rm requirements.txt

[efagerberg]

@techalchemy would you say that npm not having a virtual environment for their package locking makes it unreliable?