[BUG] [pyproject.toml] Requirements with markers are ignored if project.optional-dependencies is set

[domdfcoding]

setuptools version

setuptools==61.0.0

Python version

Python 3.8

OS

Ubuntu

Additional environment information

No response

Description

If install_requires is defined in setup.py and the [project.optional-dependencies] table is defined in pyproject.toml, requirements with markers (e.g. importlib-resources (>=3.0.0) ; python_version < "3.7") are not listed in the resulting metadata (requires.txt for sdists and METADATA for wheels).

This is due to the way setuptools handles these requirements. "Complex" requirements are added to extras_require with keys like :python_version < "3.7" and the values being the requirements without the marker (e.g. importlib-resources>=3.0.0).

setuptools/setuptools/dist.py

Lines 590 to 601 in 4923d11

	spec_inst_reqs = getattr(self, 'install_requires', None) or ()
	inst_reqs = list(_reqs.parse(spec_inst_reqs))
	simple_reqs = filter(is_simple_req, inst_reqs)
	complex_reqs = itertools.filterfalse(is_simple_req, inst_reqs)
	self.install_requires = list(map(str, simple_reqs))
	for r in complex_reqs:
	self._tmp_extras_require[':' + str(r.marker)].append(r)
	self.extras_require = dict(
	(k, [str(r) for r in map(self._clean_req, v)])
	for k, v in self._tmp_extras_require.items()
	)

However, when the metadata is loaded from pyproject.toml it is occuring after this parsing and overrides the existing value for extras_require, including these "special" keys.

Expected behavior

The resulting metadata includes all requirements listed under install_requires in setup.py, and all optional dependencies given in pyproject.toml.

How to Reproduce

1. Clone this gist: https://gist.github.com/domdfcoding/06ed6053a64b01335268c78ac1c27b04
2. Run pip wheel .
3. Check the METADATA file in the wheel. There should be the line Requires-Dist: importlib-resources (>=3.0.0) ; python_version < "3.7" but it isn't there.

Output

/tmp$ git clone https://gist.github.com/domdfcoding/06ed6053a64b01335268c78ac1c27b04
Cloning into '06ed6053a64b01335268c78ac1c27b04'...
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 7 (delta 1), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (7/7), 834 bytes | 834.00 KiB/s, done.
/tmp$ cd 06ed6053a64b01335268c78ac1c27b04
/tmp/06ed6053a64b01335268c78ac1c27b04$ pip wheel .
Processing /tmp/06ed6053a64b01335268c78ac1c27b04
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: project
  Building wheel for project (pyproject.toml) ... done
  Created wheel for project: filename=project-1.0.0-py3-none-any.whl size=1013 sha256=4d1848166743b87cb716f1af0294752bb3693ad8c6f7f9b2ca51110ed3cbedfa
  Stored in directory: /tmp/pip-ephem-wheel-cache-dbmhbh9w/wheels/16/e0/28/1173208411691c77e8cfe1f17eb35231451f2a5768eb1cc38b
Successfully built project

[abravalheri]

Thank you very much for reporting this @domdfcoding and for the investigation.

My understanding is that once we solve #3195, splitting the dependencies between pyproject.toml and setup.py would only work if dynamic includes "dependencies".