Guide request: gracefully dropping support for older Python versions

[ncoghlan]

This guide has been written by @tonybaloney and can now be found here: https://packaging.python.org/guides/dropping-older-python-versions/

---

Talking to @tonybaloney about https://twitter.com/anthonypjshaw/status/970088412177776640, I discovered that our support for gracefully dropping support for older Python versions is currently better than I realised, but it's also far from obvious how to actually do it and the potential errors to avoid.

The key moving parts are:

• the Requires-Python field in metadata 1.2+: https://packaging.python.org/specifications/core-metadata/#requires-python
• the data-requires-python link attribute in the index server release listing API: https://www.python.org/dev/peps/pep-0503/#specification
• a data-requires-python aware installer (e.g. pip 9+)
• a publishing client that allows setting the Requires-Python field in the project metadata (e.g. the python_requires setting in setuptools 24.2.0+)
• an index server that sets data-requires-python correctly (e.g. pypi.python.org since pypi/legacy@3e1ce8f)

The most conservative approach to implicitly pinning older clients to the last compatible release would be to release an alpha or development release that sets "Requires-Python: >= 3.6" (for example) in the metadata, and includes an import-time warning snippet like the one below, but doesn't actually require the new version yet:

_MINIMUM_SUPPORTED_VERSION = (3, 6, 0) # f-strings! Async generators!
if sys.version_info < _MINIMUM_SUPPORTED_VERSION:
    import warnings
    warnings.warn("This project no longer supports this version of Python, "
                  "but the release metadata indicating that has been ignored. "
                  "Please update to a newer installation client (e.g. pip 9+) and "
                  "ensure that any caching proxies in use support PEP 503's"
                  "`data-requires-python` link attribute.", FutureWarning)

(Using FutureWarning ensures that the message will be visible by default even on versions where DeprecationWarning is hidden by default)

This conservative approach has a few key benefits:

1. It allows for opt-in testing of the metadata setting by folks running pip install --pre while others on recent pip versions will ignore it even if Requires-Python didn't get set correctly (since it's a pre-release)
2. It means that folks with older deployment toolchains (even truly ancient ones that allow installation of pre-releases by default) will only get a (valid!) warning on stderr instead of a completely broken deployment.
3. It means that if something does go wrong with setting Requires-Python in the metadata, you only end up with a pre-release that emits a spurious warning rather than one that's fundamentally broken, and can easily push a new version that sets that metadata correctly (since older clients should be updated to versions that default to ignoring pre-releases).

[ncoghlan]

It occurs to me that an even more conservative approach would be to start out by specifying Requires-Python for currently supported Python versions. If you add Requires-Python: >= 2.7, != 3.0, != 3.1, != 3.2" for example, then it's relatively straightforward to get hold of a Python 2.6 installation and check that that gets pinned at the last version missing the Requires-Python` metadata as expected.

That way, the release with the warning and updated Requires-Python metadata would only be about testing how up to date user's installation tools and caching proxies are (and could potentially be skipped entirely if PyPI's download metadata indicates that folks are using up to date deployment tools even for older target environments)

[ncoghlan]

@tonybaloney has investigated this further (by way of https://github.com/tonybaloney/friendly-deprecation-test), and found that while pip 9 won't install the incompatible version on older Pythons, the mere availability of that version still prevents installation of maintenance updates for the older version - you need to explicitly amend the installation command to request a compatible version.

This means that for older versions of pip, the problem will need to be addressed at the pip-tools/pipenv level, with the tools checking for the data-requires-python metadata and amending the pinned version accordingly.

That behaviour also becomes a feature request for future versions of pip itself: checking data-requires-python compatibility when choosing the version to download, rather than when attempting to install the chosen version.

[tonybaloney]

Use case here is that package maintainers will continually drop support for out-of-support versions of Python, e.g. 2.6, 2.7 (soon), 3.2, 3.3 etc.
Every 12-18 months this will need to be updated.

Looking at the download stats of users on packages I maintain, most downloads are coming from 2.7. I'd like to drop support for 2.7-3.4 and start implementing async/await features.

But, the users of older distributions probably aren't even aware that they're installing my package, it's part of an automated script, a CI/CD process. If most of the pip install'ing happens in background processes then user warnings are going to be largely ignored and erroring completely on installation is going to break their system.

If there were instead a way to continually develop and evolve packages inline with the supported Python distributions, whilst having "legacy" releases where I can still apply security fixes.

this could be a "large version that supported Python 2.6" tag on the metadata in the package API. Which I can periodically update, following semver.

Imagining this timeline..

**July 2018: **
Release v2.0.0 of package "foobar", drops Python 2.7 and 3.3 support. Users of "foobar" installing from Python 2.7 will get a warning and pip will install the last supported version, 1.3.0 instead

August 2018:
CVE raised against package "foobar", versions v2.0.1 and v1.3.1 released. All users for Python 3.4+ will get 2.0.1 and Python 2.7 will now get v1.3.1.

September 2018
Maintainers of "foobar" want to drop Python 3.4 support and leverage asyncio, v3.0.0 is released.
Python 3.5 - v3.0.0
Python 3.4 - v2.0.1
Python 2.7, 3.3 - v1.3.1

[ncoghlan]

I think the current metadata can already express that OK - v2.0+ would have Requires-Python: >= 3.5, while v1.3.x would either continue without the setting, or would have it set to Requires-Python: >= 2.7.

The problem is that pip isn't gracefully handling that case: it's erroring out because the incompatible version exists, rather than just installing the latest compatible version.

[ncoghlan]

Take Django as an example: https://pypi.python.org/simple/django/ and https://pypi.org/simple/django/ both set data-requires-python on Django 2.0+. If pip were to simply filter out the versions with unmet Requires-Python constraints, then pip install django on Python 2.7 would automatically pick up the latest 1.x release as desired.

[ncoghlan]

cc'ing @pradyunsg and @dstufft here before I turn this into an RFE for pip's handling of data-requires-python.

[pradyunsg]

If pip were to simply filter out the versions with unmet Requires-Python constraints, then pip install django on Python 2.7 would automatically pick up the latest 1.x release as desired.

I'm not sure if I'm misunderstanding but I think this happens already.

[ncoghlan]

@pradyunsg Thanks for checking that - it seemed odd that it wasn't working that way, so I'm happy to hear the problem is somewhere else rather than in pip's handling of data-requires-python.

Your results made me realise I hadn't checked the link metadata at https://pypi.org/simple/friendly-deprecation-test/, and that shows that PyPI hasn't picked up the Requires-Python metadata correctly, which then means data-requires-python isn't getting set properly (similar to what happened with pytest in this issue report, where @ewdurbin was able to retroactively update the link metadata for the older releases in the PyPI database)

Checking https://pypi.org/project/friendly-deprecation-test/#files shows @tonybaloney hit the case where PyPI can't correctly infer data-requires-python when only an sdist is uploaded: it needs to see a wheel archive with fully rendered metadata, and it needs to see that archive before it sees the sdist for the release.

[pradyunsg]

Yeah. There's some discussion over at pypi/warehouse#474 IIRC.

Basically, certain metadata is only loaded by warehouse when a wheel is uploaded first, and it's loaded from only that wheel. This might be one of those cases.

[ncoghlan]

The specific problem hit here was to only upload the sdist (which is an entirely reasonable thing to do, it just has the consequence that PyPI doesn't know how to pull metadata from it without running arbitrary user uploaded code). I filed pypi/warehouse#3138 to suggest making it possible for project admins to set the Requires-Python metadata directly in the PyPI database.

Having two potential "sources of truth" for this info is far from ideal, but it also makes it much easier to mitigate the consequences of incorrect metadata in a release (or uploading that metadata in a way that means PyPI doesn't infer the right value for data-requires-python).

[dstufft]

I don't think it has anything to do with an sdist, you just need a new enough setuptools and twine when you do the upload.

[tonybaloney]

I was using setuptools 38.5.1 I was calling setup from distutils
and twine 1.8.1.. that'll be the issue

[tonybaloney]

@ncoghlan it works when using setuptools 👍

(env2) anthonyshaw ~/repo/friendly-deprecation-test master $ pip2 install friendly_deprecation_test --no-cache-dir
Collecting friendly_deprecation_test
  Downloading friendly_deprecation_test-1.6.0-py2-none-any.whl
Installing collected packages: friendly-deprecation-test
Successfully installed friendly-deprecation-test-1.6.0
(env2) anthonyshaw ~/repo/friendly-deprecation-test master $ pip3 install friendly_deprecation_test --no-cache-dir
Collecting friendly_deprecation_test
  Downloading friendly-deprecation-test-2.0.0.tar.gz
Installing collected packages: friendly-deprecation-test
  Running setup.py install for friendly-deprecation-test ... done
Successfully installed friendly-deprecation-test-2.0.0

[dstufft]

Oh yea, distutils doesn't support it at all.

[tonybaloney]

I did post an article about this, happy to write another explaining the process. But considering I've fumbled my way into making it work I might not be the best person to do so :-)

[ncoghlan]

@tonybaloney Thanks for working through it with us! If nobody beats me to it, I'll turn your experience with this into a packaging.python.org guide this weekend :)

[ncoghlan]

Link to @tonybaloney's article: https://hackernoon.com/phasing-out-python-runtimes-gracefully-956f112f33c4

[hugovk]

@tonybaloney Thanks for the useful article!

By the way, there's a autocorrect typo in pip install — upgrade setuptools and likewise with the smart quotes in tar xvfz dist/my-package-1.0.0.tar.gz -O | grep “Requires-Python”.

[ncoghlan]

I wrote:

I'll turn your experience with this into a packaging.python.org guide this weekend

Well, that didn't happen, and I'm no longer sure when I'll make time to do this myself (we're starting to get more feedback on the interpreter start-up changes in the 3.7 beta, so I have a few problems like https://bugs.python.org/issue33042 to find and fix).

If anyone does want to take a go at writing this themselves:

• comment here to say you're starting a draft (to reduce the change of duplicated effort)
• if you stop working on a draft, post to say that, too!
• this will be a new entry in the "guides" section of the repo
  • I suggest "Dropping Support for Older Python Versions" as the title
  • I suggest "source/guides/dropping-older-python-versions.rst" as the filename
  • I suggest listing it immediately after "supporting-multiple-python-versions" in https://github.com/pypa/python-packaging-user-guide/blob/master/source/guides/index.rst
  • adding an explicit cross-reference from the end of the multiple version support guide may also make sense (but don't worry about it for now if there doesn't seem to be a suitable place to add one)

[tonybaloney]

I'll do it today.

[tonybaloney]

Raised PR.

[ncoghlan]

#459 has been merged. Thanks again @tonybaloney, for both the original article and the guide! :)