Add API endpoint to get latest version of all projects

[dstufft]

There are projects like https://release-monitoring.org/ which want to monitor PyPI to see when new versions of specific projects are released. Currently this requires doing 1 HTTP request per tracked project which can easily turn into hundreds or thousands of HTTP requests. Offering a JSON endpoint that simply lists the names and versions of all projects can make this take a single HTTP request.

[pypingou]

\cc

[iambibhas]

It'd also be helpful if it was possible to search through the list with some filters. I've been meaning to add an instant answer to duckduckgo for python packages, but sadly it's not possible right now as there is no search API. I can also work on it. Just wondering why it doesn't exit yet.

[fungi]

Would it be possible to solve this need (the use case for release-monitoring.org) instead by streaming and parsing the PyPI update changelog, similar to how bandersnatch figures out what new updates to sync on a mirror?

[trofimander]

Would be also useful for PyCharm IDE to notify about outdated packages in the integrated package manager.

[westurner]

It could be efficient to also serve cached deltas of the whole catalog? e.g. an optional from=iso8601datetime?

• Is there something like DRPMS or rsync over HTTP that could reasonably be added in a view and generated in a celery WarehouseTask?
• etags, If-Modified-Since

[ewdurbin]

@Traff it seems that pulling current versions for all packages on the index may be a bit overzealous. getting a list from /simple should be sufficient for offering users a dialog of installable packages... the current/latest version identifier is arguably unimportant at that point.

perhaps after a user has identified a package by name, a cheap (cached at edge) call to the json API would be successful for offering a list of installable versions. GET https://pypi.python.org/pypi/<package_name>/json

regarding offering updates, again the existing json api would support a call per package to get current versions using the same call as above . these calls are cached at the edge, should be fast for end users, and are 100% fair game for community use.

i suppose my question is how is 1 long (2-10s) request to obtain a list of all package/versions better than N 200-500ms requests (which can be submitted concurrently) in order to provide information on current versions and available updates.

[ewdurbin]

to be clear, i'm not opposed to supporting the specific feature request in this issue. but am trying to aide in guiding PyCharm off of the currently used index page they are scraping for this information.

that page is incredibly expensive to generate and often causes congestion for the PyPI backends.

[trofimander]

@ewdurbin Yes, that makes sense. The only doubt we had is that we thought that making N request is worse than making one. But in this case, it could be on the contrary.

[ewdurbin]

indeed, @dstufft did a great job of summarizing the state of the world:

here and here

[westurner]

Relevant mailing list threads:

• "[Distutils] PyPI index workaround" https://mail.python.org/pipermail/distutils-sig/2016-July/029236.html
  • re: efficient ways to poll for latest versions of multiple PyPI packages

[brettcannon]

One thing like this could allow for is providing a JSON API for this info instead of having to parse the HTML of the /simple page to just get a list of projects.

[westurner]

So, what is the objective here:

• "Add API endpoint to get latest version of all projects"
  • add versions to /simple
    • cache invalidate on every package upload and then JOIN all packages
    • (so that everyone re-downloads the whole catalog every time)
• "Add API endpoint to get latest versions and package checksums of a
  specific subset of projects matching version and/or package and platform
  constraints"

On Aug 6, 2016 3:18 PM, "Brett Cannon" notifications@github.com wrote:

One thing like this could allow for is providing a JSON API for this info
instead of having to parse the HTML of the /simple page to just get a list
of projects.

So, caching and conditional requests (with etags) would probably be the
only way to afford this functionality (SELECT * FROM packages WHERE package
IN {pip, pytest, virtualenv, scripttest, mock, pretend})

Seemingly OT, but this may be the best guide to REST API cache management
(cache keys, etags, invalidation) I've ever read:
http://chibisov.github.io/drf-extensions/docs/#caching

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[wimglenn]

Hi there, just to throw in my $0.0.2 here - I use a utility which checks a requirements.txt file's pinned versions to see if there are newer releases existing on pypi. If there are N packages in the file I currently make N requests to pypi. I make a GET at, for example, https://pypi.python.org/pypi/django/json - which is a fairly large response (last time I checked, content-length: 128597) whilst the only data I'm actually interested in is under the key data['info'] to see the latest version details. Most of the response bytes are in data['releases'] which I just ignore.

It would be very helpful if there was an API endpoint which could return me just this latest version info for the package. Or, even, better for a user-specified list of packages - so I could make just one request instead of N requests. Thanks!