release_dependencies with empty string for specifier

[ewdurbin]

devpi-postgresql==0.1.1 was released this morning and was throwing 500s when trying to render on both https://pypi.python.org/pypi/devpi-postgresql/0.1.1 as well as https://pypi.org/project/devpi-postgresql/0.1.1

Investigation showed that the issue is being caused by an empty string in the specifier field for a project_url type dependency:

=> select * from release_dependencies where specifier='' and name="devpi-postgresql";
name      | devpi-postgresql
version   | 0.1.1
kind      | 8
specifier | 
id        | 670291c1-74fe-4500-906e-376439f33b09

Nothing stands out in the setup.py at devpi/devpi@b54c65c#diff-8a6f387167cc787557161cb96aad255e

@fschulze: can you specifiy what tools and versions you used to package and release devpi-postgresql==0.1.1?

I note that there is another example of an empty specifier from August... see: https://gist.github.com/anonymous/f88f6b767ade67baf8b89a2d0a2e3890

[ewdurbin]

Note: I removed the errant specifier to stop the steady flow of 500s PyPI and Warehouse were generating as a result:

=> delete from release_dependencies where id = '670291c1-74fe-4500-906e-376439f33b09';
DELETE 1

And things are all better :-D

[ewdurbin]

Warehouse Sentry Error: https://sentry.io/python-software-foundation/warehouse-production/issues/417262933/

[fschulze]

I use devpi-server which can push releases to pypi/warehouse. I recently added support for additional metadata in devpi/devpi#481, because of pytest-dev/pytest#2966.

The data currently submitted in the post on push looks like this:

{'author': 'Florian Schulze, Holger Krekel',
 'author_email': '[email protected]',
 'classifiers': ['Development Status :: 4 - Beta', 'Environment :: Web Environment', 'Intended Audience :: Developers', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: MIT License', 'Programming Language :: Python', 'Topic :: Internet :: WWW/HTTP', 'Programming Language :: Python :: Implementation :: PyPy', 'Topic :: Internet :: WWW/HTTP :: WSGI :: Application', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3.3'],
 'comment': '',
 'description': 'devpi-postgresql: a PostgreSQL storage backend for '
                'devpi-server\n'
                '===============================================================\n'
                '\n'
                '.. warning::\n'
                '    This plugin is considered experimental!\n'
                '\n'
                'This plugin adds a PostgreSQL storage backend for '
                '`devpi-server`_.\n'
                '\n'
                '.. _devpi-server: http://pypi.python.org/pypi/devpi-server\n'
                '\n'
                '\n'
                'Installation\n'
                '------------\n'
                '\n'
                '``devpi-postgresql`` needs to be installed alongside '
                '``devpi-server``.\n'
                '\n'
                'You can install it with::\n'
                '\n'
                '    pip install devpi-postgresql\n'
                '\n'
                '\n'
                'Usage\n'
                '-----\n'
                '\n'
                'When using the PostgreSQL storage, ``devpi-server`` expects '
                'an empty database.\n'
                'You have to create one like this: ``createdb devpi``\n'
                'Depending on your PostgreSQL setup you have to create a '
                'user and grant it permissions on the new database like '
                'this::\n'
                '\n'
                '    CREATE ROLE devpi WITH LOGIN;\n'
                '    GRANT CREATE, CONNECT ON DATABASE devpi TO devpi;\n'
                '\n'
                'Upon first initialization of ``devpi-server`` use '
                '``--storage pg8000`` to select the PostgreSQL backend.\n'
                '\n'
                "By default it'll use the ``devpi`` database on "
                '``localhost`` port ``5432``.\n'
                'To change that, use ``storage '
                'pg8000:host=example.com,port=5433,database=devpi_prod``.\n'
                'The possible settings are: ``database``, ``host``, '
                '``port``, ``unix_sock``, ``user`` and ``password``\n'
                '\n'
                'All user/index files and metadata of ``devpi-server`` are '
                'stored in the database.\n'
                'A few things and settings are still stored as files in the '
                'directory specified by ``--serverdir``.\n'
                '\n'
                "Plugins like ``devpi-web`` don't or can't use the storage "
                'backend.\n'
                'They still handle their own storage.\n'
                '\n'
                '\n'
                'Changelog\n'
                '=========\n'
                '\n'
                '0.1.1 (2017-12-12)\n'
                '------------------\n'
                '\n'
                '- Add ``unix_sock``, ``user`` and ``password`` to possible '
                'connection options.\n'
                '  Thanks to Szabolcs Rumi for the patch.\n'
                '\n'
                '\n'
                '\n',
 'download_url': '',
 'home_page': 'http://doc.devpi.net',
 'keywords': '',
 'license': 'MIT',
 'maintainer': '',
 'maintainer_email': '',
 'name': 'devpi-postgresql',
 'obsoletes': [],
 'obsoletes_dist': [],
 'platform': [],
 'project_urls': '',
 'provides': [],
 'provides_dist': [],
 'requires': [],
 'requires_dist': ['devpi-server (>=3.0.0.dev2)', 'pg8000'],
 'requires_external': [],
 'requires_python': '',
 'summary': 'devpi-postgresql: a PostgreSQL storage backend for devpi-server',
 'supported_platform': [],
 'version': '0.1.1'}

Should I just filter out anything with empty values?

It would then look like this:

{'author': 'Florian Schulze, Holger Krekel',
 'author_email': '[email protected]',
 'classifiers': ['Development Status :: 4 - Beta', 'Environment :: Web Environment', 'Intended Audience :: Developers', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: MIT License', 'Programming Language :: Python', 'Topic :: Internet :: WWW/HTTP', 'Programming Language :: Python :: Implementation :: PyPy', 'Topic :: Internet :: WWW/HTTP :: WSGI :: Application', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3.3'],
 'description': 'devpi-postgresql: a PostgreSQL storage backend for '
                'devpi-server\n'
                '===============================================================\n'
                '\n'
                '.. warning::\n'
                '    This plugin is considered experimental!\n'
                '\n'
                'This plugin adds a PostgreSQL storage backend for '
                '`devpi-server`_.\n'
                '\n'
                '.. _devpi-server: http://pypi.python.org/pypi/devpi-server\n'
                '\n'
                '\n'
                'Installation\n'
                '------------\n'
                '\n'
                '``devpi-postgresql`` needs to be installed alongside '
                '``devpi-server``.\n'
                '\n'
                'You can install it with::\n'
                '\n'
                '    pip install devpi-postgresql\n'
                '\n'
                '\n'
                'Usage\n'
                '-----\n'
                '\n'
                'When using the PostgreSQL storage, ``devpi-server`` expects '
                'an empty database.\n'
                'You have to create one like this: ``createdb devpi``\n'
                'Depending on your PostgreSQL setup you have to create a '
                'user and grant it permissions on the new database like '
                'this::\n'
                '\n'
                '    CREATE ROLE devpi WITH LOGIN;\n'
                '    GRANT CREATE, CONNECT ON DATABASE devpi TO devpi;\n'
                '\n'
                'Upon first initialization of ``devpi-server`` use '
                '``--storage pg8000`` to select the PostgreSQL backend.\n'
                '\n'
                "By default it'll use the ``devpi`` database on "
                '``localhost`` port ``5432``.\n'
                'To change that, use ``storage '
                'pg8000:host=example.com,port=5433,database=devpi_prod``.\n'
                'The possible settings are: ``database``, ``host``, '
                '``port``, ``unix_sock``, ``user`` and ``password``\n'
                '\n'
                'All user/index files and metadata of ``devpi-server`` are '
                'stored in the database.\n'
                'A few things and settings are still stored as files in the '
                'directory specified by ``--serverdir``.\n'
                '\n'
                "Plugins like ``devpi-web`` don't or can't use the storage "
                'backend.\n'
                'They still handle their own storage.\n'
                '\n'
                '\n'
                'Changelog\n'
                '=========\n'
                '\n'
                '0.1.1 (2017-12-12)\n'
                '------------------\n'
                '\n'
                '- Add ``unix_sock``, ``user`` and ``password`` to possible '
                'connection options.\n'
                '  Thanks to Szabolcs Rumi for the patch.\n'
                '\n'
                '\n'
                '\n',
 'home_page': 'http://doc.devpi.net',
 'license': 'MIT',
 'name': 'devpi-postgresql',
 'requires_dist': ['devpi-server (>=3.0.0.dev2)', 'pg8000'],
 'summary': 'devpi-postgresql: a PostgreSQL storage backend for devpi-server',
 'version': '0.1.1'}

[ewdurbin]

@fschulze thanks! having that POST contents is super helpful!

This issue is best classified as a validation issue rather than a client issue. I suspect that the empty string value for project_urls is what's causing the issue, but will get a test together to confirm.

[ewdurbin]

@fschulze we resolved the validation. One note for devpi is that it should be sending over a list for project_urls now!

[fschulze]

See devpi/devpi@bbc4992
I cobbled that list together. Is there a canonical source for it? Maybe even something we could depend and import from?

[di]

@fschulze Sadly no, there is not a canonical source. Metadata is specified at https://packaging.python.org/specifications/core-metadata/ but the conversion between this and "what fields PyPI accepts" is not documented. Perhaps we need more information here: https://warehouse.readthedocs.io/api-reference/legacy/#upload-api