Support docs building/upload without invoking setup.py

[jaraco]

In order to wean ourselves off of an assumption of setuptools and build on the emergent PEP 517 and 518 behavior, I'm seeking to avoid invoking setup.py at all in a process to build and upload docs to devpi from a tagged commit in CI.

I see in #388 and #546, I and others have encountered this issue before.

The comments in #546 imply that it should be possible for a routine to build the docs and then upload them.

I've got a routine that does this in a tox env:

	python -m sphinx docs html
	python -m zipfile -c dist/docs.zip html/*
	devpi use {env:TWINE_REPOSITORY_URL}
	devpi upload --only-docs dist/docs.zip

But it fails on the last line with

docs.zip: does not contain PKGINFO, skipping

[fschulze]

The zip file currently needs a filename in the following form:
{pkgname}-{version}.doc.zip

I think we can remove the requirement for the version with this patch:

diff --git a/client/devpi/upload.py b/client/devpi/upload.py
index d6ae6881..e28a6eb0 100644
--- a/client/devpi/upload.py
+++ b/client/devpi/upload.py
@@ -236,8 +236,10 @@ def get_name_version_doczip(basename):
     DOCZIPSUFFIX = ".doc.zip"
     assert basename.endswith(DOCZIPSUFFIX)
     fn = basename[:-len(DOCZIPSUFFIX)]
-    name, version = name_version_regex.match(fn).groups()[:2]
-    return name, version
+    m = name_version_regex.match(fn)
+    if m:
+        return m.groups()[:2]
+    return (fn, None)

I only tested this with one quick upload.

[fschulze]

If there is no version in the doczip name, then the documentation is assigned to the latest version of any released packages.

[jaraco]

That makes sense. I'll have to figure out if it's possible to determine the package name without invoking setup.py (pypa/packaging-problems#224).

[fschulze]

@jaraco do you think the above patch should go into devpi? Or would you just add the version to the filename once you can get the metadata for the package name anyway?

[jaraco]

@fschulze I don't think the patch is necessary, mostly because it's only a partial measure. What I'd really like to work toward is a native docs builder in the client that builds the docs without invoking setup.py... so that if someone invokes upload --with-docs or upload --only-docs, it performs the same routine as what's happening in the original post (build docs using sphinx directly, create the zip file using metadata obtained without invoking setup.py, upload), because direct invocation of setup.py _anything_ is going to become increasingly non-viable.

I'll continue to explore ways to get the metadata, which is the main blocker for this issue.

[fschulze]

Original issue seems to be solved.

[con-f-use]

@fschulze I noticed that when I upload documentation as {pkgname}-{version}.doc.zip and upload tox tests as well, that devpi then doesn't display the metadata anymore. I suspect it tires to read that from the ...doc.zip file erroneously and fails.

[con-f-use]

That bug form my comment above, might be some sort of a special situation, because I uploaded the toxresults manually, and my version of the package does not have a source distribution, just a single wheel:

$ devpi getjson /root/test/qda_rest | jq -r '[ .result[] | .["+links"][] | .href ]'
[
  "http://subdomain.domain.top/root/test/+f/61a/1f8cbad7c619e/qda_rest-0.0.200503162154-py2.py3-none-any.whl",
  [...]
  "http://subdomain.domain.top/root/test/+f/4a7/553320be8376a/qda_rest-0.2.211125135803-py2.py3-none-any.whl",
  "http://subdomain.domain.top/root/test/+f/3bd/c3afa95a59aa9/qda-rest-0.2.211125135803.doc.zip"
]

$ curl -i --user qa:qa -H "content-type: application/json" --data-binary "@./toxreport.json" "http://subdomain.domain.top/root/test/+f/4a7/553320be8376a/qda_rest-0.2.211125135803-py2.py3-none-any.whl"
HTTP/1.1 200 OK
Server: nginx
X-Devpi-Server-Version: 5.2.0
Expires: Sat, 27 Nov 2021 00:37:33 GMT
Cache-Control: no-cache

{
  "result": "root/test/+f/4a7/553320be8376a/qda_rest-0.2.211125135803-py2.py3-none-any.whl.toxresult-20211127003729-0",
  "type": "toxresultpath"
}

However, this is still clearly a bug, because before I uploaded the extra documentation, the metadata and description from the wheel was displayed, and it is again, when I delete qda-rest-0.2.211125135803.doc.zip.

Should I open a separate issue?

[fschulze]

@con-f-use yes, please open a new issue and try to add simple instructions on how to reproduce it on a new index.

[con-f-use]

yeah, as is the custom, it doesn't happen on a clean devpi-server. great. just great. my luck exactly 🤪
I'll hold off on that issue then.

It is reproducible on the original index, however (see picture)

If you have any additional debugging ideas or ansätze, I could try on a running system, I'd be greatful @fschulze . The only difference between the two package versions, that I can spot is the presence of the documentation on the right hand side.

[alexlawriewood]

For what it's worth, we've also seen that metadata disappears as soon as documentation is uploaded, using devpi-server 6.2.0 and devpi-web 4.0.6. The text "No description in metadata" appears instead, just like con-f-use shows.

We're just building from tox, invoking devpi upload --with-docs --formats bdist_wheel, with sphinx 4.1 case it's relevant.