Finalize TLS1.0/1.1 removal on PyPI

[alex]

We've been conducting a series of brownouts to make sure people are aware of the need to make sure their clients are up to date. This issue is to track the current schedule of updates to the brownout window (which may change as time goes on):

• March 22nd, brownouts begin :00-:10 every hour
• March 25th, :00-:15 every hour
• March 28th, :00-:15, :30-:35 every hour
• March 31st, :00-:15, :30-:40 every hour
• April 3rd, :00-:15, :30-:45 every hour
• April 6th, :00-:20, :30-:50 every hour
• April 8th, TLS 1.0/1.1 disabled
• April 10th, TLS1.0/1.1 disabled at fastly handshake

[dstufft]

Just bumped the brownout to the first 15 minutes.

[jvanasco]

FYI, I got locked out for well over an hour with a pip 9.0.1 client (no proxy, traffic from NYC).

Would it be possible to provide a (temporary) whitelisted migration package to assist in upgrades, like offering pip by a name like 'pip-brownout' during this time?

It's a bit of a pain distributing and installing a downloaded .tgz of pip onto Test and CI servers (multiple virtualenvs and the system version need to be installed).

Also as a suggestion: it would make sense to scatter the brownout window as some automated systems might not try to call pip/etc until the 2nd half of the hour.

[alex]

@jvanasco for issues with the brownout, please file bugs on https://github.com/pypa/packaging-problems/issues. This bug is just for tracking the ops side of it.

[jvanasco]

@alex will do. you may want to change the message on https://status.python.org/incidents/629scscfypvt , which is hotlinked from pipi, that redirects here.

[dstufft]

I think Ernest was doing more load testing earlier, which also has the effect of mandating TLSv1.2

…

Sent from my iPhone

On Mar 25, 2018, at 6:15 PM, Jonathan Vanasco ***@***.***> wrote: @alex will do. you may want to change the message on https://status.python.org/incidents/629scscfypvt , which is hotlinked from pipi, that redirects here. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[brainwane]

Sometime between now and April 8th, could we have a few days where the brownout starts at the half-hour or includes the half-hour? I'm thinking about India, where (I presume) some substantial chunk of our traffic originates, and where timezones are a half-hour off from most other timezones.

[alex]

The next milestone will be 20 minutes outage per hour, maybe we should do :00-:10 and :30-:40?

[dstufft]

We can do that, it's pretty easy.

[pradyunsg]

Maybe OP can be updated -- to mention the :00-:10 and :30-:40 plan?

[alex]

I've updated. @dstufft of note, I switched from :00-:10 + :30-:40 to :00-:15 + :30-:35, which is the same number of minutes of brownout, but avoids any "regressions" in coverage.

[brainwane]

@alex just to be clear, since you added this to the launch/redirect milestone -- you believe that completing this is a blocker to the redirect? (So, even if we finish all the other Warehouse features/infra in that milestone before April 8th, we should wait for this?) I'm fine with it if that's your assessment, but I just want to check.

[alex]

Redirecting to warehouse for downloads has the effect of requiring TLS1.2 so we can't cut over unitl we're ready to drop 1.0/1.1.

…

On Wed, Mar 28, 2018, 8:56 AM Sumana Harihareswara ***@***.***> wrote: @alex <https://github.com/alex> just to be clear, since you added this to the launch/redirect milestone <https://github.com/pypa/warehouse/milestone/1> -- you believe that completing this is a blocker to the redirect? (So, even if we finish all the other Warehouse features/infra in that milestone before April 8th, we should wait for this?) I'm fine with it if that's your assessment, but I just want to check. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3411 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAADBMMpAiJZ7Iz2g1wwk9Iv_nbapq6tks5ti4h_gaJpZM4S6IvR> .

[dstufft]

Brownout updated for :00-:15, :30-:35 every hour

[dstufft]

Brownout updated to :00-:15, :30-:40 every hour.

[dstufft]

Brownout updated to :00-:15, :30-:45 every hour.

[ben-albrecht]

From the PyPI status page:

For users of pip on MacOS/OS X, upgrading to the latest pip should resolve the issue. pip users on other platforms should upgrade their OpenSSL to a version which supports TLSv1.2.

Out of curiosity, what about MacOS/OS X is unique such that users only need to update pip?

[alex]

@ben-albrecht macOS users are impacted because the system Python linked against an ancient copy of OpenSSL that macOS shipped until 10.13. We workaround this in really recent versions of pip by using SecureTransport, which is the official Apple TLS library, and it supported TLS1.2 a long time ago.

[ben-albrecht]

@alex - Thanks for the info!

I have observed that easy_install is also vulnerable to this brownout on OS X. I assume this is because easy_install uses the system OpenSSL on OS X instead of SecureTransport like pip does - is that right?

[alex]

Yup, we recommend simply not using easy_install - it has many many deficiencies besides this one, pip is the installer of the present and future.

…

On Thu, Apr 5, 2018 at 5:06 PM, Ben Albrecht ***@***.***> wrote: @alex <https://github.com/alex> - Thanks for the info! I have observed that easy_install is also vulnerable to this brownout on OS X. I assume this is because easy_install uses the system OpenSSL on OS X instead of SecureTransport like pip does - is that right? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3411 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAADBD_h6uO-SL62oJ6NvaFmCEnCiN5Dks5tlodXgaJpZM4S6IvR> .

-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: D1B3 ADC0 E023 8CA6

[dstufft]

Brownout updated to April 6th, :00-:20, :30-:50 every hour.

2 more days left till this is complete!

[dstufft]

Brownout has been updated to a blackout, all TLSv1.0 and TLSv1.1 access to PyPI now results in a HTTP 403 error:

$ curl -i https://pypi.python.org/simple/ --tlsv1.0 --http1.1
HTTP/1.1 403 TLSv1.2+ is required
Server: Varnish
Retry-After: 0
Content-Type: text/plain; charset=UTF-8
Content-Length: 170
Accept-Ranges: bytes
Date: Sun, 08 Apr 2018 16:07:40 GMT
Via: 1.1 varnish
Connection: close
X-Served-By: cache-ewr18134-EWR
X-Cache: MISS
X-Cache-Hits: 0
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Permitted-Cross-Domain-Policies: none

Support for TLSv1.0 has been removed, please upgrade to a TLSv1.2+ client. Please see https://pyfound.blogspot.com/2017/01/time-to-upgrade-your-python-tls-v12.html

In 2 more days, we will disable TLSv1.0 and TLSv1.1 completely at the TLS handshake level.

[ncoghlan]

Is there a good end-user reference on exactly what they need to cope with the TLS v1.0/v1.1 support? https://pyfound.blogspot.com.au/2017/01/time-to-upgrade-your-python-tls-v12.html is the best I found, and that's a little dated now.

Context is pypa/packaging-problems#137, where I'm not sure what's going on with Mac OS X, and neither the PSF blog post, nor the Python 2.7 documentation are clear on exactly which python.org installers will work, and which Mac OS X system Python versions will work.

[dstufft]

I think all of the Python.org installers for 2.7 are currently affected. Actually uploading didn't occur to me at all, it would be reasonable to raise an issue with Twine to see if we can get a quick release out that enables SecureTransport on macOS.

We don't have a great end user guide, but effectively they need a Python that links against a new enough version of OpenSSL (python -c "import ssl; print(ssl.OPENSSL_VERSION)" needs to say 1.0.1 or higher, unless their distro has patched OpenSSL to add/remove TLSv1.2 support without updating version). The other way around it is to use a library other than OpenSSL to handle TLS, for requests/urllib3 on macOS this is as simple as:

import sys, ssl

# Checks for OpenSSL 1.0.1 on MacOS
if sys.platform == "darwin" and ssl.OPENSSL_VERSION_NUMBER < 0x1000100f:
    try:
        import urllib3.contrib.securetransport
    except (ImportError, OSError):
        pass
    else:
        urllib3.contrib.securetransport.inject_into_urllib3()

The above snippet of code only uses SecureTransport in cases where it's needed.

[dstufft]

Ned Deily went into some pretty fantastic detail (although it was pip specific, but pip specific really just means it has the above snippet or not) in #3293 (comment).

[ncoghlan]

Ah, I thought I'd seen something from @brainwane about this: https://mail.python.org/pipermail/python-announce-list/2018-April/011885.html

For whatever reason, Google still isn't indexing mail.python.org very well, so even highly relevant posts often don't show up in search results :(

[dstufft]

Email sent to Fastly to get TLSv1.0 and TLSv1.1 disabled, will update and close when they've done that.

[dstufft]

This is done now.

$ curl -i https://pypi.python.org/simple/ --tlsv1.0 --http1.1
curl: (35) error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version

$ curl -i https://pypi.org/simple/ --tlsv1.0 --http1.1
curl: (35) error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version