ENH: add support for PEP 639

[dnicolodi]

Closes gh-270

[henryiii]

Looks pretty good from a quick review. I'll keep this in mind when I implement it (very soon) in scikit-build-core.

[dnicolodi]

Looks pretty good from a quick review. I'll keep this in mind when I implement it (very soon) in scikit-build-core.

It is a bit uglier than what I would like because I wanted to keep compatibility with older pyproject-metadata. meson-python supports anything back to version 0.7.1.

[QuLogic]

I gave this a shot in matplotlib/matplotlib#28982, and License-Expression is correctly pulled from meson.build, as are License-File from the main project, but what I'd be missing there is License-File from the subprojects.

[dnicolodi]

I'm not very well versed in licensing issues, but I don't think the way PEP 639 supports declaring package licenses works well for this use case. The main problem is that it expects the license of the package's sdist and wheel to be exactly the same. For packages like matplotlib that link statically with some libraries, this is not the case: the license of the source code is the matplotlib license, the license of the wheel is the union of the matplotlib license with the license of all the libraries linked to it.

Concretely, the issue is that in the most common case the subprojects directory does not contain the source code of the subprojects but only references to them (in the matplotlib case in the form of meson wrap files). However, the License-File fields need to point to actual files distributed in the sdist and in the wheel. The sdist in these cases does not contain the license files, thus the License-File fields for the subprojects cannot be added. The metadata information for the sdist and the wheel need to be the same, thus the fields cannot be added to the wheel metadata either.

I don't know how the problem could be solved. The most formally right solution would to allow source and binary distributions to have different licenses. But this is a quite significant change in behavior for the package metadata system.

The pragmatic solution is to add the subprojects' license files to the source distribution and accept that the sdist as a whole might have a more restrictive license than what it could have.

[eli-schwartz]

However, the License-File fields need to point to actual files distributed in the sdist and in the wheel. The sdist in these cases does not contain the license files, thus the License-File fields for the subprojects cannot be added. The metadata information for the sdist and the wheel need to be the same, thus the fields cannot be added to the wheel metadata either.

This is not true. Metadata can be dynamic and vary between the sdist and the wheel. It's already dynamic as a result of being calculated from meson introspection, so you just need to mark that in the sdist as well.

The decision to make sdist metadata static and forbidden to change is... definitely a decision of all time. But the good news is it comes with an escape hatch. This escape hatch is needed since the spec also says that it's illegal per the spec to apply open source patches to an sdist metadata if the fields are marked as static. 🤷

[eli-schwartz]

Dynamic metadata is seen as such a nice feature that there is a discussion about eliminating it: https://discuss.python.org/t/brainstorming-elliminating-dynamic-metadata/71405/1

I don't have the time to read these long threads on Python discourse (even more so because discourse is horrible) let alone contribute any opinions. But I thought that some here may be interested...

Agreed on both counts about reading the Python discourse.

But from a quick look at the initial post before the site became impossible to read, it appears to make a few basic arguments about why dynamic metadata is "bad".

• users of editable installs have inaccurate versions from git describe
• poetry and uv rerun metadata generation constantly and would rather not have to execute a build backend hook to find out things like the version or dependencies, as it is "slow"

I find both arguments weak enough that I suspect the dislike of dynamic metadata came first and the supporting arguments were created to fit the narrative. None of this matters for sdists or wheels because both contain static metadata explicitly intended to be a cache, and editable installs are an extremely specific use case that users have already chosen to make some sacrifices in order to use -- plainly they are not worried about the version number or if they care they are using myprog --version which isn't actually covered by pypa standards about dynamic vs static metadata.

Lockfiles can just avoid self-inflicted wounds by generating them when asked, but it doesn't actually matter either way since lockfiles only have to do ANY of this on bare source trees, not dependencies obtained via sdist/ wheel, which means that users who use poetry / uv can choose to avoid dynamic metadata and users who prefer dynamic metadata can generate lockfiles less frequently.

This is yet another example of people who think the point of a standard is to hard-line forbid people from doing perfectly reasonable things that are optional, because the people doing the forbidding don't personally value that thing, and cannot abide the idea that anyone else would value it.

This contrasts with the fact that the standard currently allows all those perfectly reasonable things precisely because they are perfectly reasonable, and forbidding them would upset the people who are perfectly justified in doing perfectly reasonable things.

The whole argument is undoubtedly just perl code. They'll get tired of it after a while and nothing will change, not even opinions.

[dnicolodi]

Ready to merge I think - but I'd hold off until packaging and PyPI have support.

packaging releases a version with the license validation and normalization. PyPI is ironing out the kinks in their support for PEP 639. I've tried to upload to PyPI a package using PRP 639 fields, but neither twine nor uv publish support Metadata 2.4 at the moment. I don't know of anyone working on adding support to twine. I wonder what is the recommended way to upload to PyPI nowadays...

[dnicolodi]

I've looked into what it would take to add support for metadata 2.4 to twine, and the answer is worst than what I expected: it would take a miracle to have it any time soon. twine uses pkginfo to parse the package metadata. So adding support for metadata 2.4 requires a new release of pkginfo. However, twine is not compatible with the latest pkginfo release. There is a pull request to make twine compatible with the latest pkginfo but it is open since June and it stalled.

I don't know why twine uses pkginfo and not packaging to parse the metadata. warehouse uses packaging thus I would have expected twine to use it too. pkginfo looks to be a one man show maintained on Launchpad in a Bazar repository https://code.launchpad.net/pkginfo

[agriyakhetarpal]

In the meantime, it is possible to install pkginfo>=1.11 after you install twine (i.e., by breaking the dependency tree) and twine check --strict should then work with wheels with the Metadata-Version set as 2.4. PyPI now accepts those wheels, too, if uploaded manually – zig-pypi (https://pypi.org/project/ziglang/0.13.0.post1/#files) is an example.

[dnicolodi]

@agriyakhetarpal The dependecy problem is not a problem in itself. It is only a problem because implementing metadatya 2.4 support in twine requires adding support to pkginfo and make twine consume the added metadata fields. If there is no maintainer interest in supporting a version of pkginfo release a long while back, I don't see how there could be interest in merging patches to work with a future release. It is my understanding that, without support in the upload tool, the license information recorded in the License-Expression metadata field is not picked up by PyPI. For example, the ziglang release you link to does not show any license information on the PyPI page. Getting accurate information shown on PyPI is the purpose of this whole exercise, thus I don't think that what you show is a successful example.

[dnicolodi]

For what is worth, uv publish can upload metadata 2.4 distributions, but the lack of explicit metadata 2.4 support means that the license is not recorded in the PyPI index and not shown in the package page on PyPI.

[rgommers]

twine is still the PyPA-recommended tool, but I expect that uv publish will replace it soon enough if twine is too slow to add support for new features like PEP 639. The uv issue with license metadata has an issue for it (unblocked, the linked bug in Warehouse is fixed already): astral-sh/uv#9204

[dnicolodi]

I've opened a bug offering to port twine from pkginfo to packaging to parse metadata and offering to do the work pypa/twine#1178 That should make supporting new metadata versions much easier. Let's see how that is received

[agriyakhetarpal]

For what is worth, uv publish can upload metadata 2.4 distributions, but the lack of explicit metadata 2.4 support means that the license is not recorded in the PyPI index and not shown in the package page on PyPI.

I understand the state of things better now, thanks for the context!

[frenzymadness]

Do you have a plan to move this forward? I see the PR for twine has already been merged.

[dnicolodi]

This is ready to merge (minus the merge conflict) and can probably be merged. However, there isn't yet a twine release supporting metadata 2.4, thus cutting a meson-python release supporting PEP 639 and emitting metadata 2.4 seems premature.

[rgommers]

Very close though, which is great to see!

[frenzymadness]

Twine 6.1.0 is out for more than a month now. Would it make sense to release meson-python as well?

[henryiii]

Scikit-build-core 0.11.0 includes PEP 639 support. Hatchling uses METADATA 2.4 now. So FWIW, I think it would be a good idea. ;)

(PS: there's a bug if your LICENSE file contains form feeds and you don't use PEP 639 and you are using Python older than 3.12.8, will be releasing a new pyproject-metadata with a fix soon. But I don't think it affects this at all.)

[dnicolodi]

There is a lengthy discussion on the Python Discourse regarding issues similar to the one discussed here regarding how to express the license of distribution that embed multiple projects either via vendoring, subprojects, or embedding shared libraries into wheels: https://discuss.python.org/t/pep-639-round-3-improving-license-clarity-with-better-package-metadata/53020/196

I haven't read all the posts (usual caveats about the structure of exchanges on Discourse and the dynamics of the Python Discourse apply) but the discussion does not seem to be going anywhere. However, who has interest in the topic (@rgommers for NumPy and SciPy, @QuLogic for Matplotlib, maybe @henryiii for his work on scikit-core) may want to have a look.

[henryiii]

Is license-files supported? I'm trying out cookie and meson-python is failing because the LICENSE file, specified as license-files = ["LICENSE"], is not in the SDist anywhere.

Ah, this isn't in a release, got it.

On the topic above, I think the partially dynamic metadata PEP will help, and there's further ideas now on the thread above.

[rgommers]

However, who has interest in the topic (@rgommers for NumPy and SciPy, @QuLogic for Matplotlib, maybe @henryiii for his work on scikit-core) may want to have a look.

Oh yes, thanks for pointing that out. I somehow didn't get notifications. This is a problem for complex projects. I have only read the first ~10 posts, but we've been dealing with this problem in NumPy and SciPy for a long time in hacky ways like dynamically amending license files when cibuildwheel runs. PEP 639 is an improvement, but doesn't completely solve this issue due to ambiguities between "project license" and "project + vendored stuff in various build scenarios".

It shouldn't require a change in meson-python or block us from releasing support.

[dnicolodi]

I have only read the first ~10 posts

I stopped reading the thread after the first 100 posts or so went in circles trying to define whether the intent of the PEP was for the metadata to capture the source code license or the distribution artifact license...

It shouldn't require a change in meson-python or block us from releasing support.

Indeed. Nothing to do yet on our side.