Container registry fails with duplicate key (HTTP 500)

[Chief-Detektor]

Description

Hi!
I cannot push to the integrated docker registry anymore.
It's basically #21822 but I could not reopen that issue.

I use postgres as database and tried to fix the issue with the workaround provided in the ticket above. Without any success.
To be honest, I need to have a working solution and I will switch to something else. I just want to let you know that this issue still exists.
If I can help you by providing the database dump, I can do that.

Gitea Version

1.20.5

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

https://gist.github.com/Chief-Detektor/7b26fc63faa5bd84b5ce942fe51b6d74

Screenshots

No response

Git Version

2.40.1

Operating System

linux

How are you running Gitea?

I use the official helm chart.

Database

PostgreSQL

[Chief-Detektor]

Ok I did some research:

The error occurs also on sqlite so the database is not the problem.
I tested all versions down to 1.19.4 by checking the tags out and build them locally.
Every version would give a ....manifests/latest: 404 Not Found error which indicated to me that the issue is not directly related to gitea's registry implementation!

Out of curiosity I built and push the same Docker image from an other machine running docker 20.10.21 and that works.

Sooo I conclude that the source of my error is actually the way my docker client on macos (version 24.0.6) is pushing the images.
This is good and bad news at the same time, since it seems that only newer clients are affected and I don't know if it is even possible to determine the client's version during the push.

I have some time to investigate further and I will post my findings here.

[KN4CK3R]

It may also be helpfull if you could upload the log of all requests done by the docker client until the error happens.

[Chief-Detektor]

I found a way to reproduce the issue:
Setting the --provenance=false flag during image build resolves the issue. I can then push images without any issue.
So that narrows the problem down to that additional data that is produced/pushed to the registry.
Here is the log: without the provenance flag and after with provenance disabled.

[Chief-Detektor]

I made some debugging on version v1.20.5 and the error occurs here.
The internal error code is "MANIFEST_BLOB_UNKOWN"

The data that can't be processed is this:

"{\n  \"schemaVersion\": 2,\n  \"mediaType\": \"application/vnd.oci.image.index.v1+json\",\n  \"manifests\": [\n    {\n      \"mediaType\": \"application/vnd.oci.image.manifest.v1+json\",\n      \"digest\": \"sha256:24b84a625efffabedfbb53b55989ef6acde42b818031c066488e7c3133eb304f\",\n      \"size\": 1997,\n      \"platform\": {\n        \"architecture\": \"arm64\",\n        \"os\": \"linux\"\n      }\n    },\n    {\n      \"mediaType\": \"application/vnd.oci.image.manifest.v1+json\",\n      \"digest\": \"sha256:6a67b5ef6547ef3837fcb7418ef640d4714179e6fb79af"

[Chief-Detektor]

When using podman as client the image can be built and pushed without any errors.
Note: docker is here the alias for podman

docker build -t <domain>/chiefdetektor/nginxtest . --push

as well as

docker build -t <domain>/chiefdetektor/nginxtest . --push --provenance=false

work.

The only difference is there are more entries in the package view on the admin site:

Detail view of the latest version:

[phaynes]

Well I am glad I finally found this post - I was about to abandon gitea and just use distribution and raw git.

While provenance is important in my use case, it would be good in the short term if the container documentation includes the line --push --provenance=false so it is clear.

It took ages to resolve whether the fault was in my HTTPS proxy, compose setup, gitea setup or file permissions issue.
When I did the smoke test against distribution, the push worked straight out of the box.

[oxbee]

Faced similar issue described in #31215. Can confirm that it relates to MacOS docker client.