Add a "reference" section pragmatically summarising interoperability standards

[ncoghlan]

As per the thread at https://mail.python.org/pipermail/distutils-sig/2015-April/026206.html, it would be desirable to have stable homes on packaging.python.org that documented:

• the current version of various interoperability specifications
• which versions of the PyPA tools introduced support for that version of the specification
• end user (as opposed to tool developer) focused overviews of the specification's usage

[qwcode]

re-pasting my comment from the thread:

the PyPA site has a PEP reference that includes details on implementation:

https://www.pypa.io/en/latest/peps

I don't think we need another reference in the Packaging User Guide (PUG). We could mention that the PyPA one exists in the PUG.

As for user-facing PEP docs, I think the docs for PUG/pip/setuptools/wheel etc.. should handle that inline, and act as the layer that references (if need be) the correct PEP that applies to the feature being described.

For example, the PUG references PEP440 in a few places.

Ideally, it should be a requirement to update the docs for the major projects (and the PUG) before releasing a PEP-implementations into those projects.

For example, a round of docs PRs prior to the release of PEP440 into pug & pip could have likely prevented the confusion over the meaning of ">" that unfortunately had to be sorted out after the release.

[ncoghlan]

I'm more talking about "What problem does this specification solve?" kinds of documentation, as that's something that tends to confuse end users a bit. PEP 440, for example, is the combined superset of all supported versioning schemes, and represents PyPA being opinionated on behalf of package installers to constrain the practices of package publishers.

It would also serve as an index of "core tooling docs that need reviewing and potentially updating" when we revise one of the covered specifications.

[ncoghlan]

I'd also expect the PUG reference entries to be topic based rather than driven by the PEP numbers. Looking at https://github.com/pypa/interoperability-peps for inspiration, likely topics would be:

• Dependency management
• Versions & version comparisons
• Platform compatibility markers
• Distribution archive formats
• Package installation database
• Ensuring integrity of distribution
• Build system invocation

Those topic headings would remain stable, even as their underlying PEPs were revised or superseded.

[qwcode]

I like the idea of refactoring the index in the PyPA docs by topic., just not sure about a centralized topic index like this in the PUG. I'm more inclined to cover these topics as they naturally occur in the various places in the user docs. For example, as a user, I would expect (and want) dependency management to be covered in the pip install reference (https://pip.pypa.io/en/latest/reference/pip_install.html)

[ncoghlan]

Having the reference docs on pypa.io would potentially work, but part of my goal here is to build awareness of packaging.python.org as the site for authoritative guidance on the Python packaging ecosystem. Out of the ~30 people at the packaging open space at PyCon only a handful had even heard of packaging.python.org, and two of those were me and Steve Dower.

The other reason I'd like the reference pages to be in PyPUG is to provide a discoverability path from there to the deeper reference docs and development efforts. Even if most users don't dig that deep, I'd like to enable the possibility.

[dstufft]

I think it makes sense to have these on packaging.python.org as they aren't specific to how one tool does anything so they don't naturally belong in any one tools documentation. Taking PEP 440 for example, we should have some sort of documentation, designed for end users (which PEP 440 itself isn't), that all the various tools can point at and go "We support the standard version numbers whose documentation is here: XXXX".

On the other hand, it's possible that the right place for that documentation to live is in the packaging library, however I think it's better for it to live on packaging.python.org because most end users are not going to be aware of what that library is, that we use it, but they are going to know they want to deal with version numbers.

I do think we don't need to try to inclusively contain every single piece of PEP we write for it, but instead just the ones that have impact on end users who aren't implementing tooling to deal with things. For implementers I think it's totally fine to tell them to go read PEPs and to understand the rationalization behind them but I don't think it's reasonable to expect people to read a several thousand page treatise on the history of Python version numbering, why we choose particular syntaxes. They just want to know how to write a version number.

[ncoghlan]

As Donald suggests, the aim would be to provide "the bare minimum you need to know about ".

For PEP 440, that would be examples of a couple of valid ways of numbering versions, without delving into the complexities of everything the full spec ended up allowing in order to get us up to 99.7% compatibility with existing packages.

For 426, I expect we'll eventually want a user facing overview of the different kinds of dependencies, the extras system, and the conditional dependencies system.

[dstufft]

We can also be more opinionated about things in packaging.python.org than we can in an interoperability spec too. Not too opinionated of course, but we could outline one or more concrete suggested version schemes that fit within PEP 440. Semver-like being an obvious one as well as possibly date based for instance.

[ncoghlan]

Right, while PEP 440 gives some examples (https://www.python.org/dev/peps/pep-0440/#examples-of-compliant-version-schemes), I didn't talk about the typical semantics of "major.minor.maintenance" numbering schemes at all, and semantic versioning was mentioned mainly in terms of syntactic compatibility: https://www.python.org/dev/peps/pep-0440/#compatibility-with-other-version-schemes

[ncoghlan]

Well, I guess I wrote about compatible releases in https://www.python.org/dev/peps/pep-0440/#compatible-release, but the two sections aren't specifically linked, whereas a user focused guide would talk more about how the choice of numbering scheme relates to the use of ~= to select versions later.

[ncoghlan]

Separately from this idea, pypa/pypa.io#11 covers moving the specifications themselves to a subsection of PyPUG.

Depending on how that works out, we may decide this separate section wouldn't be worth the additional documentation maintenance overhead.

[ncoghlan]

With https://packaging.python.org/specifications/ now giving each of the core specifications its own page, I'm going to declare this implemented - with the presentation of the specs themselves separated from the PEP process that defines them, we're free to make non-functional readability changes through the normal PyPUG PR process.

This means that just like the language reference in the CPython docs, the rationale for changes can be kept in the PEPs, while the interoperability specs themselves can focus on readability for tool developers and end users.