Commands not available when using --system-site-packages

[blueyed]

I am trying to setup a virtualenv with --system-site-packages to use compiled/pre-built packages, and noticed that e.g. ipython fails to run with:

% pyenv virtualenv --system-site-packages 3.4.2 paperwork3
...
% ipython
pyenv: ipython: command not found

The `ipython' command exists in these Python versions:
...

Trying to install it skips it, because it appears to be installed already.

Could pyenv somehow handle this, especially after trying to (re-)install it?

[randy3k]

I have the same issue with pip.

[randy3k]

any thoughts?

[joaoponceleao]

This is strange. I am suddenly getting this as well. Used to work fine (minus updates of homebrew python modules).
What is also strange, is that after doing a pyenv virtualenv --system-site-packages system something, pyenv virtualenvs claims it was created in /usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4, and it does indeed create a python 3 virtualenv instead of a python2 (which is the system python).

[joaoponceleao]

Just thought I'd mention I solved both problems above (wrong system python and pip etc not found) by installing in the following way:
pyenv virtualenv --system-site-packages --python=/usr/local/bin/python yourvirtualenv
obviously you should change to xx/python3 if thats the version you're going for

[randy3k]

Thanks, it works for me. Wonder why it is not the default.

pyenv virtualenv --system-site-packages --python=`pyenv which python` venv

edit, it did not solve the problem. I can run pip now, but all third party binary files are missing.

[bobby-chiu]

Even though I installed something by pyenv virtualenv --system-site-packages --python=/usr/local/bin/python myvirtualenv, and then did pyenv rehash, ipython still fails to run.
The only way I can solve this issue is to copy ipython command from python directory path/to/.pyenv/versions/*/bin/ipython. But after enter ipython environment, it throws warning: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
It bored me. Any solutions?

[joaoponceleao]

@randy3k I would not use --python=pyenv which python. Its better to be explicit about the python path you want if you're having issues with pyenv. I've learnt not to trust the pyenv which python command lately.

@bobby-chiu I would undo those copies. It seems you got ipython confused by copying the binary into every single pyenv version (not just virtualenv). It should not be necessary to copy modules directly to the python directory (the only case I can think this is needed is with the vips image bindings). I would then uninstall that virtualenv and start again. In both cases, always make sure that the system is seeing the correct python (by running python directly and by doing a simple pip list just to make sure) before running that install command. Of course, you should also check that the python you are trying to do a virtualenv from is actually working properly beforehand. Also, make sure that the pyenv path is being called before anything else in your profiles (and not being overridden somewhere at the bottom of that file - you might think it is, but there are many different profiles in the system that may compete with each other - its all in the pyenv FAQ: put pyenv at the top of each unless you have a reason not to).

I haven't had any issues since. Its quite stable. The only issues I can see anyone having is the fact that pyenv does not compile python with shared libraries by default (this will affect certain modules, but not ipython), but that is covered elsewhere in the forum.

[bobby-chiu]

@joaoponceleao I have reinstalled both pyenv and virtualenv, and checked python and pyenv path, but it is still not worked. Actually, all the exetuable commands are unable to be inheried by --system-site-packages argument when creating new environment. Seen from $PATH, standalone virtualenv version will insert path/to/.pyenv/versions/*/bin in front of path/to/.pyenv/shims. Is this the reason? I also doubt what actions will be pyenv rehash done for exetuable commands?

[joaoponceleao]

Sorry @bobby-chiu but I don't think I can be much more help. Pyenv is rock-solid (except when it breaks, in which case it collapses entirely - it likes to be on either extreme).

I will say the following though.
In my system, .pyenv/versions/ will be the first path if I'm inside a pyenv virtualenv, otherwise its .pyenv/shims/, so I don't think thats a problem.
Normally, when using virtualenvs with system-site-packages, people tend to keep their main python with a minimum number of packages. In my case, ipython is installed inside a pyenv virtualenv that has the system-site-packages argument, its not installed in the main python. Considering the warning generated by ipython, I would install it inside the virtualenv.

You seem to insinuate that nothing from the system python works inside the virtualenv, but otherwise everything is normal? If thats the case its a bit strange, and maybe its time for the developer to get involved. @yyuu any thoughts?

[yyuu]

@joaoponceleao Sorry for delay. At least for now, pyenv/pyenv-virtualenv won't work perfectly for python executables installed in system even if it is configured to use a virtualenv created with --system-site-packages. It would work as,

• it's able to find libraries installed in system
• it's unable to find executables installed in system

As a workaround for latter case, I'd recommend to set system as a secondary version in pyenv. If there're multiple versions are set in pyenv, it'll fallback if an executable isn't installed in primary version. It'll mitigates the friction.

$ pyenv virtualenv --system-site-packages system foo
$ pyenv shell foo system

[yyuu]

It might be better to install a pyenv exec hook from pyenv-virtualenv to fallback to use system executables if activated version is a virtualenv and it was created with --system-site-packages. The hook impl will be like https://github.com/yyuu/pyenv-which-ext. This is just an idea and I have not yet implemented any of lines, though.

[jessesanford]

I am still seeing this issue in 20151006. I can see from the commit that 21239c9 made it into that tag. Am I doing something wrong?

$ pyenv virtualenv --system-site-packages 3.5.0 tornado
Ignoring indexes: https://pypi.python.org/simple
Requirement already satisfied (use --upgrade to upgrade): setuptools in /Users/jessesanford/.pyenv/versions/3.5.0/lib/python3.5/site-packages
Requirement already satisfied (use --upgrade to upgrade): pip in /Users/jessesanford/.pyenv/versions/3.5.0/lib/python3.5/site-packages
$ cd foo
pyenv-virtualenv: deactivate default
pyenv-virtualenv: activate tornado
$ pip list                                                                            
pyenv: pip: command not found

The `pip' command exists in these Python versions:
  2.7.10
  3.5.0
  default

[yyuu]

@therealjessesanford umm. plz let me know some information for investigation.

1. output of 'PYENV_VERSION=3.5.0 pyenv virtualenv --virsion.'
2. list of files in '{pyenv root}/versions/tornado/lib/*'

[jessesanford]

$ PYENV_VERSION=3.5.0 pyenv virtualenv --version                                                                                                                                           
pyenv-virtualenv 20151006 (pyvenv 3.5.0)
$ tree -L 4 ~/.pyenv/versions/tornado/lib                                                                                                                                                   
/Users/jessesanford/.pyenv/versions/tornado/lib
└── python3.5
    └── site-packages
        ├── Tornado_JSON-1.2.0-py3.5.egg-info
        │   ├── PKG-INFO
        │   ├── SOURCES.txt
        │   ├── dependency_links.txt
        │   ├── installed-files.txt
        │   ├── requires.txt
        │   └── top_level.txt
        ├── __pycache__
        │   └── easy_install.cpython-35.pyc
        ├── _markerlib
        │   ├── __init__.py
        │   ├── __pycache__
        │   └── markers.py
        ├── dill
        │   ├── __diff.py
        │   ├── __init__.py
        │   ├── __pycache__
        │   ├── _objects.py
        │   ├── detect.py
        │   ├── dill.py
        │   ├── info.py
        │   ├── objtypes.py
        │   ├── pointers.py
        │   ├── settings.py
        │   ├── source.py
        │   └── temp.py
        ├── dill-0.2.4-py3.5.egg-info
        │   ├── PKG-INFO
        │   ├── SOURCES.txt
        │   ├── dependency_links.txt
        │   ├── installed-files.txt
        │   ├── not-zip-safe
        │   └── top_level.txt
        ├── easy_install.py
        ├── jsonschema
        │   ├── __init__.py
        │   ├── __main__.py
        │   ├── __pycache__
        │   ├── _format.py
        │   ├── _reflect.py
        │   ├── _utils.py
        │   ├── _validators.py
        │   ├── _version.py
        │   ├── cli.py
        │   ├── compat.py
        │   ├── exceptions.py
        │   ├── schemas
        │   ├── tests
        │   └── validators.py
        ├── jsonschema-2.5.1.dist-info
        │   ├── DESCRIPTION.rst
        │   ├── METADATA
        │   ├── RECORD
        │   ├── WHEEL
        │   ├── entry_points.txt
        │   ├── metadata.json
        │   ├── pbr.json
        │   └── top_level.txt
        ├── pip
        │   ├── __init__.py
        │   ├── __main__.py
        │   ├── __pycache__
        │   ├── _vendor
        │   ├── basecommand.py
        │   ├── baseparser.py
        │   ├── cmdoptions.py
        │   ├── commands
        │   ├── compat
        │   ├── download.py
        │   ├── exceptions.py
        │   ├── index.py
        │   ├── locations.py
        │   ├── models
        │   ├── operations
        │   ├── pep425tags.py
        │   ├── req
        │   ├── status_codes.py
        │   ├── utils
        │   ├── vcs
        │   └── wheel.py
        ├── pip-7.1.2.dist-info
        │   ├── DESCRIPTION.rst
        │   ├── METADATA
        │   ├── RECORD
        │   ├── WHEEL
        │   ├── entry_points.txt
        │   ├── metadata.json
        │   ├── pbr.json
        │   └── top_level.txt
        ├── pkg_resources
        │   ├── __init__.py
        │   ├── __pycache__
        │   └── _vendor
        ├── setuptools
        │   ├── __init__.py
        │   ├── __pycache__
        │   ├── archive_util.py
        │   ├── cli-32.exe
        │   ├── cli-64.exe
        │   ├── cli-arm-32.exe
        │   ├── cli.exe
        │   ├── command
        │   ├── compat.py
        │   ├── depends.py
        │   ├── dist.py
        │   ├── extension.py
        │   ├── gui-32.exe
        │   ├── gui-64.exe
        │   ├── gui-arm-32.exe
        │   ├── gui.exe
        │   ├── lib2to3_ex.py
        │   ├── msvc9_support.py
        │   ├── package_index.py
        │   ├── py26compat.py
        │   ├── py27compat.py
        │   ├── py31compat.py
        │   ├── sandbox.py
        │   ├── script\ (dev).tmpl
        │   ├── script.tmpl
        │   ├── site-patch.py
        │   ├── ssl_support.py
        │   ├── unicode_utils.py
        │   ├── utils.py
        │   ├── version.py
        │   └── windows_support.py
        ├── setuptools-18.2.dist-info
        │   ├── DESCRIPTION.rst
        │   ├── METADATA
        │   ├── RECORD
        │   ├── WHEEL
        │   ├── dependency_links.txt
        │   ├── entry_points.txt
        │   ├── metadata.json
        │   ├── top_level.txt
        │   └── zip-safe
        ├── tornado
        │   ├── __init__.py
        │   ├── __pycache__
        │   ├── auth.py
        │   ├── autoreload.py
        │   ├── concurrent.py
        │   ├── curl_httpclient.py
        │   ├── escape.py
        │   ├── gen.py
        │   ├── http1connection.py
        │   ├── httpclient.py
        │   ├── httpserver.py
        │   ├── httputil.py
        │   ├── ioloop.py
        │   ├── iostream.py
        │   ├── locale.py
        │   ├── locks.py
        │   ├── log.py
        │   ├── netutil.py
        │   ├── options.py
        │   ├── platform
        │   ├── process.py
        │   ├── queues.py
        │   ├── simple_httpclient.py
        │   ├── speedups.cpython-35m-darwin.so
        │   ├── stack_context.py
        │   ├── tcpclient.py
        │   ├── tcpserver.py
        │   ├── template.py
        │   ├── test
        │   ├── testing.py
        │   ├── util.py
        │   ├── web.py
        │   ├── websocket.py
        │   └── wsgi.py
        ├── tornado-4.2.1-py3.5.egg-info
        │   ├── PKG-INFO
        │   ├── SOURCES.txt
        │   ├── dependency_links.txt
        │   ├── installed-files.txt
        │   └── top_level.txt
        └── tornado_json
            ├── __init__.py
            ├── __pycache__
            ├── api_doc_gen.py
            ├── application.py
            ├── constants.py
            ├── exceptions.py
            ├── gen.py
            ├── jsend.py
            ├── requesthandlers.py
            ├── routes.py
            ├── schema.py
            └── utils.py

[jessesanford]

However maybe I am doing it all wrong? I assumed that --system-site-packages switch allows me to avoid having to specify both the "tornado" env and the "3.5.0" env (that it is based on) in my .python-version file. Is that correct? What is the purpose of --system-site-packages if I still need to have them both in that file?

[yyuu]

Your adsumption is correct. '--system-site-packages' is a feature of virtualenv, and it makes you to avoid having multiple versions in your .puthon-version. Something is not working at least on your installation, though.

[yyuu]

Umm. It looks like the hook I prepared in 21239c9 isn't working well with virtual environment created with pyvenv --system-site-packages. Now I'm looking for workaround for it.

Meanwhile, using virtualenv instead of pyvenv would work for you. In that case, just installing virtualenv by PYENV_VERSION=3.5.0 pip install virtualenv would force pyenv virtualenv to use virtualenv over pyvenv.

[yyuu]

@therealjessesanford Pushed 521576e. This should work with virtual environment created with pyvenv --system-site-packages. Please give it a try.

[jessesanford]

Sorry for the delay. I just installed the homebrew HEAD version which I imagine would contain this latest patch and I am still having trouble:

jessesanford@JesseMacbook ~/workspace/gke-jenkins-docker (master●)$ pyenv --version
pyenv 20151124
jessesanford@JesseMacbook ~/workspace/gke-jenkins-docker (master●)$ pyenv virtualenv --version 
pyenv-virtualenv 20151103 (virtualenv unknown)
jessesanford@JesseMacbook ~/workspace/gke-jenkins-docker (master●)$ pyenv virtualenv --system-site-packages 3.5.0 qutebrowser
Ignoring indexes: https://pypi.python.org/simple
Requirement already satisfied (use --upgrade to upgrade): setuptools in /Users/jessesanford/.pyenv/versions/3.5.0/lib/python3.5/site-packages
Requirement already satisfied (use --upgrade to upgrade): pip in /Users/jessesanford/.pyenv/versions/3.5.0/lib/python3.5/site-packages
pyenv-virtualenv: deactivate
pyenv-virtualenv: activate qutebrowser
pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior.
(qutebrowser) jessesanford@JesseMacbook ~/workspace/gke-jenkins-docker (master●)$ pip install pyqt5 
pyenv: pip: command not found

The `pip' command exists in these Python versions:
  2.7.10
  3.5.0
  default
  tornado

[jessesanford]

I installed as a plugin by doing the git clone into the ~/.pyenv/plugins folder and things are working now. However it definitely does not work when using any version of pyenv-virtualenv installed with homebrew.

[yyuu]

Umm. I fixed a problem in installation script in c61fdaa. brew installation should work now.

[impguard]

So, installing virtualenv resolves the issue in #139. pip freeze -l correctly shows only locally installed packages.

However, I'm still hitting this issue, where the ipython command is not found. The same error:

pyenv: ipython: command not found

The `ipython' command exists in these Python versions:
  3.5.0

This occurs in the same conditions as the examples in this issue.

[yyuu]

It's just working even with virtual environments created with venv at least for me.

% ls ~/.pyenv/versions/3.5.1/bin
2to3@                  f2py3.5*   jp.py*                jupyter-notebook*   pydoc@                  pyrsa-keygen*      python3.5m*         rst2man.py*
2to3-3.5*              idle@      jsonschema*           jupyter-qtconsole*  pydoc3@                 pyrsa-priv2pub*    python3.5m-config*  rst2odt_prepstyles.py*
aws*                   idle3@     jupyter*              jupyter-trust*      pydoc3.5*               pyrsa-sign*        python3-config@     rst2odt.py*
aws.cmd*               idle3.5*   jupyter-console*      pbr*                pygmentize*             pyrsa-verify*      python-config@      rst2pseudoxml.py*
aws_completer*         iptest*    jupyter-kernelspec*   pip@                pyrsa-decrypt*          python@            pyvenv@             rst2s5.py*
aws_zsh_completer.sh*  iptest3*   jupyter-migrate*      pip3*               pyrsa-decrypt-bigfile*  python3@           pyvenv-3.5*         rst2xetex.py*
easy_install@          ipython*   jupyter-nbconvert*    pip3.5*             pyrsa-encrypt*          python3.5*         rst2html.py*        rst2xml.py*
easy_install-3.5*      ipython3*  jupyter-nbextension*  __pycache__/        pyrsa-encrypt-bigfile*  python3.5-config@  rst2latex.py*       rstpep2html.py*
% ls ~/.pyenv/versions/venv35-global/bin
activate  activate.csh  activate.fish  python@  python3@  python3.5@
% cat ~/.pyenv/versions/venv35-global/pyvenv.cfg
home = /home/yyuu/.pyenv/versions/3.5.1/bin
include-system-site-packages = true
version = 3.5.1
% PYENV_VERSION=3.5.1 ipython
Python 3.5.1 (default, Dec 17 2015, 00:43:09) 
Type "copyright", "credits" or "license" for more information.

IPython 4.0.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
UsageError: Invalid GUI request 'inline', valid ones are: glut, osx, wx, tk, qt5, qt, gtk3, gtk, pyglet
In [1]: 

[impguard]

It works with venv. But venv has the problem where pip freeze --local fails.

Virtualenv resolves that issue, and by itself actually works well (freezing works and binaries work). Pyenv-virtualenv with the virtualenv backing doesn't seem to work for me.

[yyuu]

@impguard I confirmed the behaviour, and fixed it in 4c8eeeb. Give it a try.

[impguard]

@yyuu I've installed pyenv through homebrew. Hope this isn't a dumb question, but how do I check out a specific commit this way?

[impguard]

Pulled in latest release. Works as intended now.

[erny]

Got the same issue creating virtualenv with pyenv virtualenv --system-site-packages myenv from the default (global) version which is also a pyenv installed version.