Skip to content

"executable py33/bin/python3.3 is not functioning" with mac framework build #322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bdarnell opened this issue Sep 1, 2012 · 7 comments
Closed

Comments

@bdarnell
Copy link

bdarnell commented Sep 1, 2012

It looks like virtualenv and python 3.3's built-in venv support interact badly in a mac framework build. Virtualenv creation fails with the message "executable not functioning". I've traced through it enough to figure out what's happening:

  1. "virtualenv -p python3.3" re-execs itself under /opt/local/bin/python3.3
  2. In a framework build, $prefix/bin/pythonX.Y is a "pythonw" executable, which sets __PYVENV_LAUNCHER__ then executes the "real" binary (Resources/Python.app/Contents/MacOS/Python). The real binary uses the launcher environment variable (if present) to determine sys.prefix
  3. virtualenv copies the real binary (not pythonw) into the environment as $env/bin/pythonX.Y
  4. When virtualenv runs the env's python to test it, the launcher variable from the pythonw process that started virtualenv is still present, so the env's python uses the global value for sys.prefix.
  5. If the env's python is run in a clean environment (without __PYVENV_LAUNCHER__), it finds the right prefix.

It looks like virtualenv needs to ensure that the env's python is always run with a correct (or missing) __PYVENV_LAUNCHER__ variable, and never inherits one from the environment. One way to do this would be to copy the pythonw shim into $env/bin instead of the underlying python, but I don't know what other ramifications this would have.

Even though virtualenv is not needed as much since 3.3 has equivalent functionality built in, it's still important that virtualenv work with 3.3 because tools like tox do not yet support 3.3-native venvs.

My environment: OSX Mountain Lion, python 3.3rc1 installed via macports, virtualenv installed from the "develop" branch today.

(venv_tip)[bdarnell@gonzo ~]$ virtualenv -v -p python3.3 py33
Running virtualenv with interpreter /opt/local/bin/python3.3
Creating py33/lib/python3.3
Symlinking Python bootstrap modules
  Symlinking py33/lib/python3.3/config-3.3m
  Symlinking py33/lib/python3.3/lib-dynload
  Symlinking py33/lib/python3.3/os.py
  Ignoring built-in bootstrap module: posix
  Symlinking py33/lib/python3.3/posixpath.py
  Cannot import bootstrap module: nt
  Symlinking py33/lib/python3.3/ntpath.py
  Symlinking py33/lib/python3.3/genericpath.py
  Symlinking py33/lib/python3.3/fnmatch.py
  Symlinking py33/lib/python3.3/locale.py
  Symlinking py33/lib/python3.3/encodings
  Symlinking py33/lib/python3.3/codecs.py
  Symlinking py33/lib/python3.3/stat.py
  Cannot import bootstrap module: UserDict
  Cannot import bootstrap module: copy_reg
  Symlinking py33/lib/python3.3/types.py
  Symlinking py33/lib/python3.3/re.py
  Cannot import bootstrap module: sre
  Symlinking py33/lib/python3.3/sre_parse.py
  Symlinking py33/lib/python3.3/sre_constants.py
  Symlinking py33/lib/python3.3/sre_compile.py
  Cannot import bootstrap module: _abcoll
  Symlinking py33/lib/python3.3/warnings.py
  Symlinking py33/lib/python3.3/linecache.py
  Symlinking py33/lib/python3.3/abc.py
  Symlinking py33/lib/python3.3/io.py
  Symlinking py33/lib/python3.3/_weakrefset.py
  Symlinking py33/lib/python3.3/copyreg.py
  Symlinking py33/lib/python3.3/tempfile.py
  Symlinking py33/lib/python3.3/random.py
  Symlinking py33/lib/python3.3/__future__.py
  Symlinking py33/lib/python3.3/collections
  Symlinking py33/lib/python3.3/keyword.py
  Symlinking py33/lib/python3.3/tarfile.py
  Symlinking py33/lib/python3.3/shutil.py
  Symlinking py33/lib/python3.3/struct.py
  Symlinking py33/lib/python3.3/copy.py
  Symlinking py33/lib/python3.3/tokenize.py
  Symlinking py33/lib/python3.3/token.py
  Symlinking py33/lib/python3.3/functools.py
  Symlinking py33/lib/python3.3/heapq.py
  Symlinking py33/lib/python3.3/bisect.py
  Symlinking py33/lib/python3.3/weakref.py
  Symlinking py33/lib/python3.3/reprlib.py
  Symlinking py33/lib/python3.3/base64.py
  Symlinking py33/lib/python3.3/_dummy_thread.py
  Symlinking py33/lib/python3.3/hashlib.py
  Symlinking py33/lib/python3.3/hmac.py
  Symlinking py33/lib/python3.3/rlcompleter.py
Creating py33/lib/python3.3/site-packages
Writing py33/lib/python3.3/site.py
Writing py33/lib/python3.3/orig-prefix.txt
Writing py33/lib/python3.3/no-global-site-packages.txt
Creating parent directories for py33/include
Symlinking py33/include/python3.3m
Creating py33/bin
New python executable in py33/bin/python3.3
Changed mode of py33/bin/python3.3 to 0o755
Also creating executable in py33/bin/python
Changed mode of py33/bin/python to 0o755
Symlinking py33/.Python
Testing executable with py33/bin/python3.3 -c "import sys;out=sys.stdout;getattr(out, "buffer", out).write(sys.prefix.encode("utf-8"))"
ERROR: The executable py33/bin/python3.3 is not functioning
ERROR: It thinks sys.prefix is '/opt/local/Library/Frameworks/Python.framework/Versions/3.3' (should be '/Users/bdarnell/py33')
ERROR: virtualenv is not compatible with this system or executable
@pnasrat
Copy link

pnasrat commented Sep 2, 2012

Reproduced locally - thanks for the detailed report.

@pnasrat
Copy link

pnasrat commented Sep 2, 2012

This gets further with the below diff but fails with

OSError: Command /Users/pnasrat/Devel...env/33/bin/python3.3 -x /Users/pnasrat/Devel.../33/bin/easy_install /Users/pnasrat/Devel...pport/pip-1.2.tar.gz failed with error code 2

Running with -vvv

  Extracting distribute-0.6.28-py3.3.egg to /Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages
  distribute 0.6.28 is already the active version in easy-install.pth
  Installing easy_install script to /Library/Frameworks/Python.framework/Versions/3.3/bin
  Installing easy_install-3.3 script to /Library/Frameworks/Python.framework/Versions/3.3/bin
diff --git a/virtualenv.py b/virtualenv.py
index 2316d7c..f16dcdd 100755
--- a/virtualenv.py
+++ b/virtualenv.py
@@ -1257,6 +1257,10 @@ def install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages, clear):
     mkdir(bin_dir)
     py_executable = join(bin_dir, os.path.basename(sys.executable))
     if 'Python.framework' in prefix:
+        # OS X framework builds cause validation to break
+        # https://github.com/pypa/virtualenv/issues/322
+        if os.environ.get('__PYVENV_LAUNCHER__'):
+          os.unsetenv('__PYVENV_LAUNCHER__')
         if re.search(r'/Python(?:-32|-64)*$', py_executable):
             # The name of the python executable is not quite what
             # we want, rename it.

@pnasrat
Copy link

pnasrat commented Sep 2, 2012

I've done a pull request and I'll try to follow up with a fix for the incorrect location for install of scripts.

pnasrat added a commit to pnasrat/virtualenv that referenced this issue Sep 2, 2012
Fixes pypa#322 on OS X Framework installs.
@pnasrat
Copy link

pnasrat commented Sep 2, 2012

Notes to self this works:

./33/bin/python virtualenv_embedded/distribute_setup.py -vvv --always-copy -U distribute

@pnasrat
Copy link

pnasrat commented Sep 2, 2012

Can you try with the patches on my develop branch in Pull Request #323

@bdarnell
Copy link
Author

bdarnell commented Sep 2, 2012

Yes, the changes in #323 work for me in that it creates a working virtualenv, although it doesn't solve the underlying issue that any time a non-virtualenv python3.3 process starts a virtualenv python3.3 as a subprocess this environment variable will leak through and cause the same problem. Basically, the entry points to python should not use the PYVENV_LAUNCHER variable from their environment. If Contents/MacOS/Python is an implementation detail that should only be invoked as a child of pythonw (aka $prefix/bin/python), then virtualenv should install the wrapper instead of reaching into the inner executable (it looks like this is what 3.3 venv does). If Contents/MacOS/Python is intended to be a public entry point to the interpreter, then this is really a cpython bug and the interaction between the two processes needs to be tweaked. The answer may be as simple as having cpython unset PYVENV_LAUNCHER in the environment once initialization is complete, so it won't spread to downstream processes.

@pnasrat pnasrat closed this as completed in 6793d5f Sep 3, 2012
@pnasrat
Copy link

pnasrat commented Sep 3, 2012

I'll try follow up on the subprocess chain stuff on various lists and if I need to will file another bug.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants