Skip to content

On systems with older SSL, packages that trigger easy_install fail #134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ilovezfs opened this issue Mar 31, 2018 · 12 comments
Open

On systems with older SSL, packages that trigger easy_install fail #134

ilovezfs opened this issue Mar 31, 2018 · 12 comments
Labels
pypi-tls Issues with PyPI's TLS<1.2 brownouts

Comments

@ilovezfs
Copy link

bash-3.2$ /usr/local/Cellar/gitfs/0.4.5.1_1/libexec/bin/pip --version
pip 9.0.3 from /usr/local/Cellar/gitfs/0.4.5.1_1/libexec/lib/python2.7/site-packages (python 2.7)
bash-3.2$  

but

iMac-TMP:homebrew-core joe$ brew instal -dvs gitfs
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/gitfs.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/libgit2.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/pkg-config.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/cmake.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/sphinx-doc.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/libssh2.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/openssl.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/makedepend.rb
/usr/bin/sandbox-exec -f /tmp/homebrew20180330-16347-uu47na.sb nice /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.3/bin/ruby -W0 -I /usr/local/Homebrew/Library/Homebrew -- /usr/local/Homebrew/Library/Homebrew/build.rb /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/gitfs.rb --verbose --debug
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FromPathLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/gitfs.rb
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/libgit2.rb
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/libssh2.rb
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/openssl.rb
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/pkg-config.rb
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/cmake.rb
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/sphinx-doc.rb
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/makedepend.rb
==> Downloading https://github.com/PressLabs/gitfs/archive/0.4.5.1.tar.gz
Already downloaded: /Users/joe/Library/Caches/Homebrew/gitfs-0.4.5.1.tar.gz
==> Verifying gitfs-0.4.5.1.tar.gz checksum
tar xzf /Users/joe/Library/Caches/Homebrew/gitfs-0.4.5.1.tar.gz
==> Downloading https://files.pythonhosted.org/packages/b1/72/2d70c5a1de409ceb3a27ff2ec007ecdd5cc52239e7c74990e32af57affe9/virtualenv-15.2.0.tar.gz
Already downloaded: /Users/joe/Library/Caches/Homebrew/gitfs--homebrew-virtualenv-15.2.0.tar.gz
==> Verifying gitfs--homebrew-virtualenv-15.2.0.tar.gz checksum
tar xzf /Users/joe/Library/Caches/Homebrew/gitfs--homebrew-virtualenv-15.2.0.tar.gz
==> python2.7 -c import setuptools... --no-user-cfg install --prefix=/tmp/gitfs--homebrew-virtualenv-20180330-16349-1dpywqn/target --single-version-externally-managed --record=installed.txt
running install
running build
running build_py
creating build
creating build/lib
copying virtualenv.py -> build/lib
creating build/lib/virtualenv_support
copying virtualenv_support/__init__.py -> build/lib/virtualenv_support
copying virtualenv_support/argparse-1.4.0-py2.py3-none-any.whl -> build/lib/virtualenv_support
copying virtualenv_support/pip-9.0.3-py2.py3-none-any.whl -> build/lib/virtualenv_support
copying virtualenv_support/setuptools-36.8.0-py2.py3-none-any.whl -> build/lib/virtualenv_support
copying virtualenv_support/setuptools-39.0.1-py2.py3-none-any.whl -> build/lib/virtualenv_support
copying virtualenv_support/wheel-0.29.0-py2.py3-none-any.whl -> build/lib/virtualenv_support
copying virtualenv_support/wheel-0.30.0-py2.py3-none-any.whl -> build/lib/virtualenv_support
running install_lib
copying build/lib/virtualenv.py -> /tmp/gitfs--homebrew-virtualenv-20180330-16349-1dpywqn/target/lib/python2.7/site-packages
creating /tmp/gitfs--homebrew-virtualenv-20180330-16349-1dpywqn/target/lib/python2.7/site-packages/virtualenv_support
copying build/lib/virtualenv_support/__init__.py -> /tmp/gitfs--homebrew-virtualenv-20180330-16349-1dpywqn/target/lib/python2.7/site-packages/virtualenv_support
copying build/lib/virtualenv_support/argparse-1.4.0-py2.py3-none-any.whl -> /tmp/gitfs--homebrew-virtualenv-20180330-16349-1dpywqn/target/lib/python2.7/site-packages/virtualenv_support
copying build/lib/virtualenv_support/pip-9.0.3-py2.py3-none-any.whl -> /tmp/gitfs--homebrew-virtualenv-20180330-16349-1dpywqn/target/lib/python2.7/site-packages/virtualenv_support
copying build/lib/virtualenv_support/setuptools-36.8.0-py2.py3-none-any.whl -> /tmp/gitfs--homebrew-virtualenv-20180330-16349-1dpywqn/target/lib/python2.7/site-packages/virtualenv_support
copying build/lib/virtualenv_support/setuptools-39.0.1-py2.py3-none-any.whl -> /tmp/gitfs--homebrew-virtualenv-20180330-16349-1dpywqn/target/lib/python2.7/site-packages/virtualenv_support
copying build/lib/virtualenv_support/wheel-0.29.0-py2.py3-none-any.whl -> /tmp/gitfs--homebrew-virtualenv-20180330-16349-1dpywqn/target/lib/python2.7/site-packages/virtualenv_support
copying build/lib/virtualenv_support/wheel-0.30.0-py2.py3-none-any.whl -> /tmp/gitfs--homebrew-virtualenv-20180330-16349-1dpywqn/target/lib/python2.7/site-packages/virtualenv_support
byte-compiling /tmp/gitfs--homebrew-virtualenv-20180330-16349-1dpywqn/target/lib/python2.7/site-packages/virtualenv.py to virtualenv.pyc
byte-compiling /tmp/gitfs--homebrew-virtualenv-20180330-16349-1dpywqn/target/lib/python2.7/site-packages/virtualenv_support/__init__.py to __init__.pyc
running install_egg_info
running egg_info
writing virtualenv.egg-info/PKG-INFO
writing top-level names to virtualenv.egg-info/top_level.txt
writing dependency_links to virtualenv.egg-info/dependency_links.txt
writing entry points to virtualenv.egg-info/entry_points.txt
reading manifest file 'virtualenv.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*' found under directory 'docs/_templates'
warning: no previously-included files matching '*' found under directory 'docs/_build'
writing manifest file 'virtualenv.egg-info/SOURCES.txt'
Copying virtualenv.egg-info to /tmp/gitfs--homebrew-virtualenv-20180330-16349-1dpywqn/target/lib/python2.7/site-packages/virtualenv-15.2.0-py2.7.egg-info
running install_scripts
Installing virtualenv script to /tmp/gitfs--homebrew-virtualenv-20180330-16349-1dpywqn/target/bin
writing list of installed files to 'installed.txt'
==> python2.7 -s /tmp/gitfs--homebrew-virtualenv-20180330-16349-1dpywqn/target/bin/virtualenv -p python2.7 /usr/local/Cellar/gitfs/0.4.5.1_1/libexec
New python executable in /usr/local/Cellar/gitfs/0.4.5.1_1/libexec/bin/python
Installing setuptools, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2.7
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/python.rb
==> Downloading https://files.pythonhosted.org/packages/86/8c/70aea8215c6ab990f2d91e7ec171787a41b7fbc83df32a067ba5d7f3324f/atomiclong-0.1.1.tar.gz
Already downloaded: /Users/joe/Library/Caches/Homebrew/gitfs--atomiclong-0.1.1.tar.gz
==> Verifying gitfs--atomiclong-0.1.1.tar.gz checksum
tar xzf /Users/joe/Library/Caches/Homebrew/gitfs--atomiclong-0.1.1.tar.gz
==> /usr/local/Cellar/gitfs/0.4.5.1_1/libexec/bin/pip install -v --no-deps --no-binary :all: --ignore-installed /private/tmp/gitfs--atomiclong-20180330-16349-1dq3o08/atomiclong-0.1.1
Processing /private/tmp/gitfs--atomiclong-20180330-16349-1dq3o08/atomiclong-0.1.1
  Running setup.py (path:/private/tmp/pip-jc5alv-build/setup.py) egg_info for package from file:///private/tmp/gitfs--atomiclong-20180330-16349-1dq3o08/atomiclong-0.1.1
    Running command python setup.py egg_info
    Couldn't find index page for 'cffi' (maybe misspelled?)
    No local packages or working download links found for cffi
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/tmp/pip-jc5alv-build/setup.py", line 42, in <module>
        cmdclass={"test": PyTest},
      File "/usr/local/Cellar/gitfs/0.4.5.1_1/libexec/lib/python2.7/site-packages/setuptools/__init__.py", line 128, in setup
        _install_setup_requires(attrs)
      File "/usr/local/Cellar/gitfs/0.4.5.1_1/libexec/lib/python2.7/site-packages/setuptools/__init__.py", line 123, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/usr/local/Cellar/gitfs/0.4.5.1_1/libexec/lib/python2.7/site-packages/setuptools/dist.py", line 513, in fetch_build_eggs
        replace_conflicting=True,
      File "/usr/local/Cellar/gitfs/0.4.5.1_1/libexec/lib/python2.7/site-packages/pkg_resources/__init__.py", line 774, in resolve
        replace_conflicting=replace_conflicting
      File "/usr/local/Cellar/gitfs/0.4.5.1_1/libexec/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1057, in best_match
        return self.obtain(req, installer)
      File "/usr/local/Cellar/gitfs/0.4.5.1_1/libexec/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1069, in obtain
        return installer(requirement)
      File "/usr/local/Cellar/gitfs/0.4.5.1_1/libexec/lib/python2.7/site-packages/setuptools/dist.py", line 580, in fetch_build_egg
        return cmd.easy_install(req)
      File "/usr/local/Cellar/gitfs/0.4.5.1_1/libexec/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 667, in easy_install
        raise DistutilsError(msg)
    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('cffi')
Cleaning up...
Command "python setup.py egg_info" failed with error code 1 in /private/tmp/pip-jc5alv-build/
Exception information:
Traceback (most recent call last):
  File "/usr/local/Cellar/gitfs/0.4.5.1_1/libexec/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/Cellar/gitfs/0.4.5.1_1/libexec/lib/python2.7/site-packages/pip/commands/install.py", line 335, in run
    wb.build(autobuilding=True)
  File "/usr/local/Cellar/gitfs/0.4.5.1_1/libexec/lib/python2.7/site-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/local/Cellar/gitfs/0.4.5.1_1/libexec/lib/python2.7/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/local/Cellar/gitfs/0.4.5.1_1/libexec/lib/python2.7/site-packages/pip/req/req_set.py", line 634, in _prepare_file
    abstract_dist.prep_for_dist()
  File "/usr/local/Cellar/gitfs/0.4.5.1_1/libexec/lib/python2.7/site-packages/pip/req/req_set.py", line 129, in prep_for_dist
    self.req_to_install.run_egg_info()
  File "/usr/local/Cellar/gitfs/0.4.5.1_1/libexec/lib/python2.7/site-packages/pip/req/req_install.py", line 439, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/usr/local/Cellar/gitfs/0.4.5.1_1/libexec/lib/python2.7/site-packages/pip/utils/__init__.py", line 707, in call_subprocess
    % (command_desc, proc.returncode, cwd))
InstallationError: Command "python setup.py egg_info" failed with error code 1 in /private/tmp/pip-jc5alv-build/
/usr/local/Homebrew/Library/Homebrew/debrew.rb:11:in `raise'
BuildError: Failed executing: /usr/local/Cellar/gitfs/0.4.5.1_1/libexec/bin/pip install -v --no-deps --no-binary :all: --ignore-installed /private/tmp/gitfs--atomiclong-20180330-16349-1dq3o08/atomiclong-0.1.1
1. raise
2. ignore
3. backtrace
4. irb
5. shell
Choose an action: 

CC @alex

This is running with

iMac-TMP:Homebrew joe$ git diff
diff --git a/Library/Homebrew/language/python_virtualenv_constants.rb b/Library/Homebrew/language/python_virtualenv_constants.rb
index 4efd2ce..64d2416 100644
--- a/Library/Homebrew/language/python_virtualenv_constants.rb
+++ b/Library/Homebrew/language/python_virtualenv_constants.rb
@@ -1,2 +1,2 @@
-PYTHON_VIRTUALENV_URL = "https://files.pythonhosted.org/packages/d4/0c/9840c08189e030873387a73b90ada981885010dd9aea134d6de30cd24cb8/virtualenv-15.1.0.tar.gz".freeze
-PYTHON_VIRTUALENV_SHA256 = "02f8102c2436bb03b3ee6dede1919d1dac8a427541652e5ec95171ec8adbc93a".freeze
+PYTHON_VIRTUALENV_URL = "https://files.pythonhosted.org/packages/b1/72/2d70c5a1de409ceb3a27ff2ec007ecdd5cc52239e7c74990e32af57affe9/virtualenv-15.2.0.tar.gz".freeze
+PYTHON_VIRTUALENV_SHA256 = "1d7e241b431e7afce47e77f8843a276f652699d1fa4f93b9d8ce0076fd7b0b54".freeze
iMac-TMP:Homebrew joe$ 
@dstufft
Copy link
Member

dstufft commented Mar 31, 2018

It's not obvious, but that's actually easy_install causing this error. There is likely a setup_requires in this package that's triggering setuptools to fetch some build time dependencies outside of what pip has control over.

@ilovezfs
Copy link
Author

@dstufft yet another example: https://gist.github.com/ilovezfs/ad434ccb5849584f9234f33791486468

Same problem?

@ilovezfs
Copy link
Author

That's running with this diff

iMac-TMP:homebrew-core joe$ git diff -- Formula/conan.rb 
diff --git a/Formula/conan.rb b/Formula/conan.rb
index 79b421f..a272efd 100644
--- a/Formula/conan.rb
+++ b/Formula/conan.rb
@@ -17,10 +17,10 @@ class Conan < Formula
   depends_on "pkg-config" => :build
   depends_on "libffi"
   depends_on "openssl"
-  depends_on "python"
+#  depends_on "python"
 
   def install
-    venv = virtualenv_create(libexec, "python3")
+    venv = virtualenv_create(libexec) #, "python3")
     system libexec/"bin/pip", "install", "-v", "--no-binary", ":all:",
                               "--ignore-installed", buildpath
     system libexec/"bin/pip", "uninstall", "-y", name

(I switched it to Python 3 the other day because I was running into the brownout errors and thought it was conan specific)

@dstufft
Copy link
Member

dstufft commented Mar 31, 2018

Yes, this is the relevant bit of the output:

  Running setup.py (path:/private/tmp/pip-build-D0rZQn/pylint/setup.py) egg_info for package pylint
    Running command python setup.py egg_info
    Couldn't find index page for 'pytest-runner' (maybe misspelled?)
    No local packages or working download links found for pytest-runner
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/tmp/pip-build-D0rZQn/pylint/setup.py", line 178, in <module>
        install()
      File "/private/tmp/pip-build-D0rZQn/pylint/setup.py", line 175, in install
        **kwargs)
      File "/usr/local/Cellar/conan/1.2.0/libexec/lib/python2.7/site-packages/setuptools/__init__.py", line 128, in setup
        _install_setup_requires(attrs)
      File "/usr/local/Cellar/conan/1.2.0/libexec/lib/python2.7/site-packages/setuptools/__init__.py", line 123, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/usr/local/Cellar/conan/1.2.0/libexec/lib/python2.7/site-packages/setuptools/dist.py", line 513, in fetch_build_eggs
        replace_conflicting=True,
      File "/usr/local/Cellar/conan/1.2.0/libexec/lib/python2.7/site-packages/pkg_resources/__init__.py", line 774, in resolve
        replace_conflicting=replace_conflicting
      File "/usr/local/Cellar/conan/1.2.0/libexec/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1057, in best_match
        return self.obtain(req, installer)
      File "/usr/local/Cellar/conan/1.2.0/libexec/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1069, in obtain
        return installer(requirement)
      File "/usr/local/Cellar/conan/1.2.0/libexec/lib/python2.7/site-packages/setuptools/dist.py", line 580, in fetch_build_egg
        return cmd.easy_install(req)
      File "/usr/local/Cellar/conan/1.2.0/libexec/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 667, in easy_install
        raise DistutilsError(msg)
    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pytest-runner')

You can see pip is executing the setup.py, then it starts dumping the output from setup.py (indented). The traceback you see comes from setup.py, and the error traceback is rooted in setuptools.

@ilovezfs
Copy link
Author

@dstufft any suggested workarounds for these other than the use-non-Apple-Python sledgehammer?

@dstufft
Copy link
Member

dstufft commented Mar 31, 2018

You can pre-install whatever is in setup_requires so that when setuptools hits that code path, it sees they're already available and doesn't try to fetch from PyPI. You might also consider opening an issue with setuptools and ask for a SecureTransport fallback like pip has or an option that would fallback to pyOpenSSL, that would let people pre-install pyOpenSSL to get a newer OpenSSL version.

@dstufft dstufft added the pypi-tls Issues with PyPI's TLS<1.2 brownouts label Mar 31, 2018
@RichardProsser
Copy link

I have macOS "High Sierra" (10.13.3) and we use Python 3.4.4 with a 'venv' for each project; the rather obscure error message "Could not find a version that satisfies the requirement ..." occurs within the first 15 minutes of each hour, due to the TLS 'brownouts' apparently: https://status.python.org/

When the 'pip install' command succeeds, the usual upgrade prompt appears (with a new VE):
"You are using pip version 7.1.2, however version 9.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command."

Upgrading to pip 9.0.3 resolves the 'brownout' issue I believe but I cannot perform the upgrade until after the 15 minute 'closed window' is over!

Given that the Mac itself relies on Python 2.7, is there a system-wide way to upgrade pip without having to call upon PyPi? Better, could the Python distribution include the latest version of pip? I really don't see why we should be expected to upgrade pip each time, especially when this 'brownout' policy is in place.

@brainwane
Copy link
Contributor

We recommend that people upgrade their local versions of pip using:

curl https://bootstrap.pypa.io/get-pip.py | python

I sent out a note recommending this to people on pre-High Sierra versions of macOS; I think this should work for you as well. I defer to others' assessments on this.

@njsmith
Copy link
Member

njsmith commented Apr 3, 2018

we use Python 3.4.4 [...] Better, could the Python distribution include the latest version of pip?

You're using a python release from 2015. If you upgrade to the latest 3.6 release, then it does indeed include the latest version of pip.

@benjaminjkraft
Copy link

I just filed this against setuptools as suggested. Is there a good workaround other than manually pre-installing the setup_requires packages?

@alex
Copy link
Member

alex commented Apr 10, 2018 via email

@jaraco jaraco changed the title macOS 10.11 issues during brownout even with pip 9.0.3 On systems with older SSL, packages that trigger easy_install fail Dec 13, 2018
@jaraco
Copy link
Member

jaraco commented Dec 13, 2018

Is there a good workaround other than manually pre-installing the setup_requires packages.

The best recommendation is for the project to declare its build requirements in pyproject.toml per PEP 518, which will satisfy the requirements on a pip install operation as long as pip 10 is present. Also the project should publish wheels, which will allow older pip versions to install the package without triggering builds. And in #219, I'm working on identifying/developing a tool to produce sdists from PEP 517 packages so that setup_requires isn't invoked there during the publishing step.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pypi-tls Issues with PyPI's TLS<1.2 brownouts
Projects
None yet
Development

No branches or pull requests

9 participants