Reach out to 3rd party services about new pypi.org domain

[di]

We should reach out to any 3rd party service that depends on the pypi.python.org domain and encourage them to update to pypi.org.

A list of services that currently depend on pypi.python.org:

• https://badge.fury.io/for/py (clicking the badge directs the user to pypi.python.org)
• https://pyup.io/ (PRs currently link to pypi.python.org)
• TravisCI ("can automatically release your Python package to PyPI after a successful build." May fix PyPI deployments not working when "special" chars in password travis-ci/dpl#377?)
• CircleCI (similar)
• sphinx-pypi-upload
• the cheeseshop extension to Sphinx ("easily link to PyPI packages")
• Pydoc.io (site, code; grabs PyPI packages and builds API docs)
• possibly PythonAnywhere which gives people hosted Python environments & links to projects on legacy PyPI & may run a mirror? to handle pip install
• the Debian watch folks and devscripts (including uscan) folks, such as Piotr Roszatycki, as they hit legacy PyPI (uscan, watch) and produce pypi.debian.net (example) and have asked about Warehouse supporting uscan more cleanly
• Enthought whose Canopy is a curated Python package list
• http://shields.io/ (badges)
• Anitya (see Switch to querying pypi.org instead of pypi.python.org fedora-infra/anitya#531 & below)
• libraries.io
• release-monitoring.org

[brainwane]

(Of course we shouldn't reach out to the auto-deploy-to-PyPI orgs till our new infrastructure can support the load.)

Edited to add: I talked with Dustin in IRC just now. I'm running across sites/projects that could conceivably fit under this issue, like projects that upload to PyPI or that make "my project on PyPI" widgets for your site, etc.

We decided pretty much any 3rd party that is referring to the old domain is valid for this issue, and it's fine to add to the checklist in the initial comment to keep the checklist all in one place.

[ncoghlan]

Open source release tracking services like release-monitoring.org and libraries.io will also be affected.

I filed fedora-infra/anitya#531 for Anitya, and based on an initial review, the info we'll need for that migration is:

• the basic project page relocation (which I already know: https://pypi.python.org/pypi/{name} -> https://pypi.org/project/{name})
• the JSON API relocation (that was previously a matter of appending /json to the project page reference)
• the new endpoint for the package upload RSS feed

[dstufft]

URLs should be the same or redirect to their new location, just with the domain changed.

[mgedmin]

http://shields.io/ can do PyPI badges, they might need a heads up.

[di]

@brainwane Perhaps we should put together a "Migrating from pypi.python.org to pypi.org" document for 3rd parties that answers the questions @ncoghlan has (and any other), which we can point them to when we "reach out" to them.

[ncoghlan]

@di We have https://packaging.python.org/guides/migrating-to-pypi-org/, but it's currently focused on migrations for tool configurations and interactive use.

For the lower level details of API & URL migrations, I suspect a contributor focused document in Warehouse explaining the handling of the paths below /pypi/ would make sense, and then the packaging.python.org page can add a new short subsection that links to that after explaining that replacing pypi.python.org with pypi.org is expected to "just work" for any client that correctly follows redirects.

Also providing updated answers for my own questions (based on @dstuft's advice above):

• https://pypi.org/pypi/fedmsg (with or without a trailing slash) redirects to https://pypi.org/project/fedmsg/
• https://pypi.org/pypi/fedmsg/json returns the expected JSON response directly (my mistake was switching out pypi for project in the URL path)
• https://pypi.org/pypi?%3Aaction=rss redirects to https://pypi.org/rss/updates.xml

[ncoghlan]

Huh, it looks like pypi.org honours an Accept-Encoding: gzip header, whereas pypi.python.org ignored it. That may uncover some latent defects in clients that are setting that header, but not actually handling a gzip encoded response properly (with Anitya and/or Anitya's test suite potentially being one such client).

[ncoghlan]

Turns out Anitya could handle the gzip encoding support just fine, we just needed to update some of the tests to account for the refreshed recorded responses from PyPI.

[brainwane]

@brainwane Perhaps we should put together a "Migrating from pypi.python.org to pypi.org" document for 3rd parties that answers the questions @ncoghlan has (and any other), which we can point them to when we "reach out" to them.

I agree. Started in #3153.

It would be great to have #3151 for this as well.

[jayfk]

pyup.io is notified 👍🏼

[brainwane]

So, we now have the migration guide at https://warehouse.readthedocs.io/api-reference/integration-guide/#migrating-to-the-new-pypi . I've started writing up a general "things that are changing in PyPI and packaging/distribution in general" thing to send out and am putting a text dump here. Close friends of mine are having an emergency so I need to drop things and go help them for the next few days, but other folks should feel free to pick this up, edit it, reuse it as they see fit, and get more of the checkboxes in the list above checked off.

---

I'm the project manager for the new Python Package Index (Warehouse), which is currently in pre-production at http://pypi.org/ . On the Warehouse roadmap, it looks like the full switch will happen sometime in April, so here's a heads-up about why we're switching, what's changed, and what to expect.

The legacy PyPI site at https://pypi.python.org started in the early 2000s. In recent years, users faced outages, malicious packages, and spam attacks, and the legacy codebase made it hard to maintain and even harder to develop new features.

The new PyPI has a far more modern look, and is up-to-date under the hood as well; a proper web framework (Pyramid), 100% backend test coverage, and a Docker-based development environment, make it easier for current and new developers to maintain it and add features.

Thanks to Mozilla's Open Source Support funding, developers have added many new features, overhauled infrastructure, and made steady progress towards redirecting traffic to the new site and shutting down the old one. As of the middle of last year, package releases must go through the new PyPI, and as of late February, new user account registration is only available on the new site. The full switch will include redirecting browser and pip install traffic from the old site; then, sometime in late April or early May, the legacy site will be entirely shut down.

Your site/service will probably be able to seamlessly switch to the new site, and thanks to redirects, may not have to change anything immediately. Here's a migration guide.

Some new PyPI features:

• mobile-responsive UI
• chronological release history for each project (example)
• easy-to-read project activity journal for project maintainers
• better search and filtering
• support for multiple project URLs (e.g., for a homepage and a repo)
• user-visible Gravatars and email addresses for maintainers
• no need to "register" a project before initial upload
• far better backend infrastructure, reducing the frequency of outages

Things that are going away, or already have (sometimes for policy or spam-fighting reasons), include:

• pythonhosted.com documentation hosting (Add the ability to delete "legacy" documentation #582)
• download counts visible in the API (instead, use the Google BigQuery service)
• GPG/PGP signatures for packages (still visible in the Simple Project API per PEP 503, but no longer visible in the web UI
• key management: PyPI no longer has a UI for users to manage their GPG or SSH public keys
• package maintainers being able to upload a new release via the web UI (instead, the recommended command-line tool is Twine)
• package maintainers being able to log in and update release descriptions via the web UI (to update release metadata, they need to upload a new release; see distutils-sig discussion)
• OpenID and Google auth login
• users being able to upload a package without verifying their email address with PyPI first
• HTTP access to APIs; now it's HTTPS-only

And in the works:

• PEP 541 will enable more timely package takeovers, as people get package names transferred to them after conflict resolution
• Now that PEP 566 has been approved, developers are working to get Markdown supported for README files on PyPI

For future updates, please sign up for the low-traffic PyPI announcements email list.

Thank you for integrating with PyPI, and please let us know if you have any questions or problems with the new site!

[dstufft]

JSON API https://warehouse.readthedocs.io/api-reference/json/

This isn't a new feature, legacy had it and Warehouse just mimic'd it.

[ncoghlan]

Regarding GPG signatures: one of the risks that publisher signatures defend against is a compromise of PyPI itself (whether that's a technical compromise by a third party, or malicious action by a PyPI admin). However, for them to effectively serve that purpose, signatures need to be obtained through a trusted channel that isn't PyPI. As a reference link, https://mail.python.org/pipermail/distutils-sig/2017-March/030252.html would be a reasonable one (both Donald and I have written about the topic at various points over the years, that's just the one that came up first for me in Google)

Regarding PEP 541: the requirements for declaring a package as abandoned are pretty stringent, so we don't expect the number of transfer requests to increase substantially, but we do expect them to be resolved in a more timely fashion when they do come up.

[brainwane]

I updated a bit of the comment - thanks for your feedback, Donald and Nick. Now I'll use it to send notes to some of the sites and projects listed above.

[brainwane]

Reached out to TravisCI in travis-ci/dpl#779.

I took out the bullet point about verification GPG signatures on packages & pointer to distutils-sig discussion because I saw travis-ci/dpl#727 (comment) . I looked at #25, pypa/twine#157, #1439, https://mail.python.org/pipermail/distutils-sig/2016-May/028933.html , and https://pypi.python.org/security, and as far as I can tell, Warehouse does not actually change anything regarding GPG package signatures, compared to legacy PyPI.

[brainwane]

badges/shields#1569, librariesio/libraries.io#2024.

[dstufft]

As far as GPG and PyPI goes, the delta between legacy and Warehouse is uh:

• Warehouse doesn't expose the fact a file had a signature uploaded for it in the UI.
• Warehouse doesn't have a UI for users to manage their GPG public key.

I don't think anything else has changed, and both of those are UI things so it probably doesn't need called out specifically since a lot of UI stuff changed?

[brainwane]

@dstufft when you say "Warehouse doesn't expose the fact a file had a signature uploaded for it in the UI." I see, you mean the web UI, and Ernest was referring to the Simple API. Thanks.

I do think the things you mention do need calling out because - in my experience - while "foo has moved over to another page/a sidebar" doesn't need much explicit guidance, "foo that you used to see/use is now gone" will cause confusion and support requests unless we explicitly forestall that by saying "you haven't missed it, it's actually gone now, deliberately." I'll add a brief note to my boilerplate.

[dstufft]

That works. If you want to see the difference you can look at https://pypi.python.org/pypi/pip versus https://pypi.org/project/pip/#files. The former has a (pgp) link, and the latter does not.

I might be wrong about the managing your GPG key part, I just logged into legacy to get a link for the page and I don't see it anymore. I suspect @ewdurbin or @di removed it from legacy at some point and I didn't notice (or I forgot).

That reminds me of another difference though, Warehouse no longer has a mechanism for managing your SSH key either (and PyPI still has this). The practical difference is minuscule though, because the code that utilized the user's SSH key got ripped out like 4 years ago so managing your SSH key doesn't really do anything except update an otherwise unused field in a database.

[brainwane]

mosquito/pypi-server#30, circleci.py#40, an email to CircleCI support, email to gemfury about badges, and email to PythonAnywhere sent.

[brainwane]

Emailed Paul Tagliamonte, Barry Warsaw, and Piotr Roszatycki with a heads-up and asked for it to be forwarded to the debian-python list. Email sent to Jannis Leidel and WoLpH about sphinx-pypi-upload. Email sent to Georg Brandl about the Sphinx cheeseshop extension. And edited the announcement text from my comment above and put it on the wiki at https://wiki.python.org/psf/PackagingWG/PyPIBetaAnnouncement .

Am calling this complete. We can start a new issue if we start discovering more services to alert.

[di]

@jayfk Noticed that pyup is still using pypi.python.org, anything we can do to help?

[jayfk]

@jayfk Noticed that pyup is still using pypi.python.org, anything we can do to help?

I need some kind of cloning device :D. I have the updated URLs in a local branch, just need to re-run tests and merge it in.

Thanks for keeping an eye out for this!

[pradyunsg]

We didn't reach out to https://pythonwheels.com, did we?

Project: https://github.com/meshy/pythonwheels

[di]

Hey @meshy, are you aware of this? Anything we can do to help?

[meshy]

Hey @di and @pradyunsg, thanks for reaching out!

I've been aware of the upcoming changes for a while, thanks to @hugovk. They've been a star, and have done the leg-work for me in terms of migrating the project over to using BigQuery.

I've not had the time to merge the required changes, but I hope that pythonwheels will be ready when the change comes.

Also, amazing job on Warehouse! It's awesome!

[hugovk]

You're welcome. See https://hugovk.github.io/top-pypi-packages/ for a weekly dump of the 5,000 most-downloaded packages from PyPI, as a replacement for the old top_packages.