hundreds of processes spawned

[timfelle]

After installing python 3.7.4 64 bit, i attempted to start a virtual environment in a folder. However, about a thousand python processes were started and the virtual environment did not get completed.

OS: Windows 10 Home
Code run in Cygwin 64 bit

Failing code:

mkdir test
cd test
virtualenv venv

pip list

Package Version

---

astroid 2.2.0
colorama 0.4.1
isort 4.3.9
lazy-object-proxy 1.3.1
mccabe 0.6.1
pip 19.1.1
pylint 2.3.0
rope 0.14.0
setuptools 40.8.0
six 1.12.0
typed-ast 1.3.1
virtualenv 16.6.1
wrapt 1.11.1

[AndrYast]

I tried to install python, pip and virtualenv for the first time today and faced the same problem.

I've fixed it by commenting 3 lines in
Python\Python37\Lib\site-packages\virtualenv.py
and adding
-p python
when running virtualenv

The lines i commented are 783-785:
if hasattr(sys, "_base_executable"):
print("hasattr(sys, \"_base_executable\") == yes")
return sys._base_executable

[thingselliotprograms]

@AndrYast after activating the environment are you still on 3.7.4? This definitely seems to be an issue specifically with 3.7.4, which was released yesterday (7/8/19).

[AndrYast]

@thingselliotprograms yes, running
venv\Scripts\activate
python --version
gives me
Python 3.7.4

[grundysf]

Experiencing same issue with pipenv hanging since upgrading python yesterday to 3.7.4. Underlying issue related to virtualenv. Downgraded python to 3.7.3 for now.

[gaborbernat]

Ack virtualenv latest is broken because python/cpython#14428 always sets sys._base_executable.

[akittas]

I'm getting the same issue with 3.7.4. I'm using pipenv and getting a [WinError 8] Not enough memory resources are available to process this command. I cannot create any virtualenv. Everything works great in 3.7.3.

[fireundubh]

I'm experiencing this exact issue, too, with 3.7.4.

Rolling thunder of python.exe processes exhausts memory and virtualenv creation ultimately fails.

The issue did not exist with 3.7.2.

[gaborbernat]

Maintainer here. As I said above the contract for some internal variable of CPython changed, and this causes now infinite loop on process creation for both python 3.7 and 3.8.

[fireundubh]

No idea whether this is a good idea, but I was able to create a virtual environment with this quick hack.

# virtualenv.py:783
if hasattr(sys, "_base_executable") and sys.version_info < (3, 7, 4):
    return sys._base_executable

Judging by the comments above that line, looks like you had some fun in an earlier version.

[pfmoore]

Maybe there needs to be an additional check that sys._base_executable != sys.executable. But honestly, I don't know - this seems to be continually changing in patch releases and I don't have the time to keep up with what's going on (it all seems to be related to work to support the "Windows Store" build of Python).

Maybe @zooba could comment or offer suggestions here. We're using undocumented internals, so ultimately the issue is up to us to address, but I don't know of a supported way to get the information we need, so I don't see a solution that won't be subject to potential breakage every time we get a new release of Python :-(

Ultimately, #1377 is probably the only reliable solution here.

[gaborbernat]

I think as long as we add that check we'll be fine.

[YakDriver]

Travis CI jobs running Windows and attempting to create a virtualenv also fail. Travis CI recently moved to 3.7.4. 😞

https://travis-ci.community/t/infinite-loop-of-virtualenv-windows/4139

Possibly helpful information from the Travis builds:

version: v6.2.0 https://github.com/travis-ci/worker/tree/5e5476e01646095f48eec13196fdb3faf8f5cbf7
instance: travis-ci-onion-1803-containers-1542208204-ad01dca (via amqp)
bash version 4.4.19(2)-release
Chocolatey v0.10.11
python3 v3.7.4 [Approved]
python3 has been installed.
Successfully installed pip-19.1.1
Successfully installed virtualenv-16.6.1

$ virtualenv $HOME/venv
Running virtualenv with interpreter c:\python37\python.exe
Running virtualenv with interpreter c:\python37\python.exe
Running virtualenv with interpreter c:\python37\python.exe
Running virtualenv with interpreter c:\python37\python.exe
Running virtualenv with interpreter c:\python37\python.exe
...[repeats 763 more times]...
The command "virtualenv $HOME/venv" failed and exited with 1 during .
Your build has been stopped.
MemoryError

[pfmoore]

I think as long as we add that check we'll be fine.

Won't doing that reintroduce (on 3.7.4) the issues the base_executable check was intended to fix? Admittedly those issues were far less serious, so it's likely still worth doing, but I don't think it's a full solution.

[gaborbernat]

I don't think it causes regression, does it?

[pfmoore]

I haven't tested one way or another (and won't have time to, sorry).

[frostming]

What's the conclusion here, is a fix release possible for virtualenv?

[gaborbernat]

Yes it is if anyone makes a PR with the fix 😊

[fireundubh]

Do you have a test for the regression you mentioned?

[gaborbernat]

I think we do.

[pfmoore]

The relevant test (added in #1345) is test_create_environment_from_venv - but note that it needs to be run in all of Python 3.7.2, 3.7.3 and 3.7.4 as the behaviour is different in each of those (minor) versions, and that's not something that the CI will cover, as far as I know.

[techalchemy]

Thanks for the quick fixes all around folks, I believe comparing base_executable with executable provides a functional equivalent to simply checking for the presence of the _base_executable attribute (possibly it's even safer as it's a bit more explicit)

I am curious what caused the issue in the first place though.

[gaborbernat]

You need to get the system python to create a virtual environment. Trying to create a virtual environment with either a venv or virtualenv package will not work.

Previously sys._base_executable was set on python 3.7+ only if we were not in system python. This changed with the above PR to always set (if non-system Python will be equal to sys.executable). The change simplifies built-in code for CPython (at the part of system standard library and site package discovery), hence the change, but it's a change of contract. That being said given it's a private attribute the change wasn't considered breaking so it's alright. Then again we could not get this information from another place so we were relying on this private field.

[sosadchuk]

Python 3.7.4
Having the same problem, posting our output in case it will be helpful:

Requirement already up-to-date: virtualenv in c:\users\tester\appdata\local\programs\python\python37\lib\site-packages (16.6.1)
Running virtualenv with interpreter c:\users\tester\appdata\local\programs\python\python37\python.exe
Running virtualenv with interpreter c:\users\tester\appdata\local\programs\python\python37\python.exe
Running virtualenv with interpreter c:\users\tester\appdata\local\programs\python\python37\python.exe
...
(Repeated 350 times in total)

Traceback (most recent call last):
  File "c:\users\tester\appdata\local\programs\python\python37\lib\site-packages\virtualenv.py", line 2611, in <module>
    main()
  File "c:\users\tester\appdata\local\programs\python\python37\lib\site-packages\virtualenv.py", line 814, in main
    sub_process_call = subprocess.Popen([interpreter, file] + sys.argv[1:], env=env)
  File "c:\users\tester\appdata\local\programs\python\python37\lib\subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "c:\users\tester\appdata\local\programs\python\python37\lib\subprocess.py", line 1178, in _execute_child
    startupinfo)
OSError: [WinError 8] Not enough memory resources are available to process this command

[pfmoore]

#1383 created. I would strongly recommend some manual testing before merging. I don't have 3.7.4 installed so I haven't tested the change at all, and I don't know what minor version of 3.7 CI is currently running.

Maybe some of the people reporting the issue here could test the change and confirm it fixes their issue and doesn't introduce any other breakage?

[YakDriver]

It works with Python 3.7.4 on Travis CI! I see no new issues.

This is the Travis config I used for the test (relevant portion):

    - stage: test
      os: windows
      language: shell
      env: PATH=/c/Python37:/c/Python37/Scripts:$PATH
      before_install:
        - choco install python
        # python -m pip install virtualenv
        - pip install git+https://github.com/pypa/virtualenv
        - virtualenv $HOME/venv
        - source $HOME/venv/Scripts/activate

The results (relevant parts):

Progress: Downloading python 3.7.4... 100%
python3 v3.7.4 [Approved]
python3 package files install completed. Performing other installation steps.
Installing 64-bit python3...
python3 has been installed.
Installed to: 'C:\Python37'
...
16.31s$ pip install git+https://github.com/pypa/virtualenv
Collecting git+https://github.com/pypa/virtualenv
  Cloning https://github.com/pypa/virtualenv to c:\users\travis\appdata\local\temp\pip-req-build-ceb1gi36
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Building wheels for collected packages: virtualenv
  Building wheel for virtualenv (PEP 517): started
  Building wheel for virtualenv (PEP 517): finished with status 'done'
  Stored in directory: C:\Users\travis\AppData\Local\Temp\pip-ephem-wheel-cache-kx8oezso\wheels\8d\58\76\749812a30b0b5c5cdc1b327e343711660ee5ebf51cf56d2df5
Successfully built virtualenv
Installing collected packages: virtualenv
Successfully installed virtualenv-16.6.1
46.51s$ virtualenv $HOME/venv
Using base prefix 'c:\\python37'
New python executable in C:\Users\travis\venv\Scripts\python.exe
Installing setuptools, pip, wheel...
done.
0.16s$ source $HOME/venv/Scripts/activate

[pfmoore]

It works with Python 3.7.4 on Travis CI! I see no new issues.

Excellent, thanks for confirming

os: windows

I hadn't realised Travis provided Windows environments now, that's interesting :-)

[YakDriver]

@pfmoore They emphasize that Windows is "early access" and has issues (significantly you can't use secrets).

[pradyunsg]

Yea, Windows support was rolled out just before #travisAlums stuff happened IIRC (the new parent company drastically shrunk Travis's team)

[liuriu]

I have this code(in file virtualenv.py) to fix it:
origin is:
if hasattr(sys, "_base_executable"):
and changed to:
if hasattr(sys, "_base_executable") and not os.environ.get("VIRTUALENV_INTERPRETER_RUNNING"):
by do this, would fix the loop

[gaborbernat]

Fix should be released via https://pypi.org/project/virtualenv/16.6.2/

[timfelle]

Just tested and it is indeed fixed. Thank you very much for the quick and good responses.