issue pushing image: BLOB_UNKNOWN: Manifest references unknown blob(s)

[bpmct]

envbuilder image: v1.0.3

I'm using Envbuilder and just saw this error when attempting to push an image with this template. This is the first time I tried it. Any ideas?

#2: ENTRYPOINT ["/.envbuilder/bin/envbuilder"]
#2: Taking snapshot of files...
#2: Pushing layer us-central1-docker.pkg.dev/coder-dogfood-v2/envbuilder-cache/coder-dogfood:54caa3ffffdf4ccf8167b88fb35d86690df3c33b209f38144d0c1a568e129d69 to cache now
#2: Pushing image to us-central1-docker.pkg.dev/coder-dogfood-v2/envbuilder-cache/coder-dogfood:54caa3ffffdf4ccf8167b88fb35d86690df3c33b209f38144d0c1a568e129d69
#2: Pushed us-central1-docker.pkg.dev/coder-dogfood-v2/envbuilder-cache/coder-dogfood@sha256:38521b4520703bca0f790759b278cf7897e4b855d910c5fc537a5831fbcaac4b
#2: Pushed us-central1-docker.pkg.dev/coder-dogfood-v2/envbuilder-cache/coder-dogfood@sha256:21095fa572a3c6de7369ceeb83c9a4ee8481a95652af28c5e008e83608510204
#2: 🏗️ Built image! [1m57.651985166s]
#3: 🏗️ Pushing image...
#3: Pushing image to us-central1-docker.pkg.dev/coder-dogfood-v2/envbuilder-cache/coder-dogfood
restore mount /usr/src/linux-headers-5.15.0-97-generic
restore mount /usr/src/linux-hwe-5.15-headers-5.15.0-97
restore mount /lib/modules/5.15.0-97-generic
Restored DOCKER_CONFIG to 
error: do push: failed to push to destination us-central1-docker.pkg.dev/coder-dogfood-v2/envbuilder-cache/coder-dogfood: PUT https://us-central1-docker.pkg.dev/v2/coder-dogfood-v2/envbuilder-cache/coder-dogfood/manifests/latest: BLOB_UNKNOWN: Manifest references unknown blob(s): sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1

[johnstcn]

cc @mafredri

[johnstcn]

I wonder if one of the blobs we previously pushed has been deleted due to age? Need to figure out how to reliably reproduce.

[bpmct]

related: i'm not sure if a failed push should prevent the user from entering their workspace. can this fail gracefully while also alerting devs/admins of the problem?

[johnstcn]

related: i'm not sure if a failed push should prevent the user from entering their workspace

It definitely shouldn't!

EDIT: Actually, there may situations where it's warranted. If you plan to run envbuilder in a CI/CD workflow, then you would want to know that pushing the image failed. We would need to be able to differentiate if pushing the image is a requirement or if pushing the image is a nice side-effect.

[MartinLoeper]

I am also experiencing a similar issue: error: do push: failed to push to destination harbor.harbor.svc.cluster.local/envbuilder-cache/test-repo2: PUT https://harbor.harbor.svc.cluster.local/v2/envbuilder-cache/test-repo2/manifests/latest: MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1

I wonder why this is happening with harbor. There is definitely no GC going on in my default harbor setup. Has any of you an idea what might be going on here?

[johnstcn]

I can reproduce this locally as well with registry:2 as the registry with the same Docker image as Ben above (mcr.microsoft.com/vscode/devcontainers/universal:focal).

I can also reproduce this issue with codercom/code-server:latest.

Strangely enough, I can see the blob on the registry filesystem before pushing, and it's just an empty file:

sudo zcat .registry-cache/docker/registry/v2/blobs/sha256/4f/4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1/data | hexdump -C
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400

[johnstcn]

Querying the registry v2 HTTP API locally, I can see that it is indeed missing despite the layers being on disk:

curl -I http://localhost:5000/v2/cache/blobs/sha256%3A4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1

would return a HTTP 404 Not Found response.

After manually uploading the blob via cURL:

curl -v -XPOST http://localhost:5000/v2/cache/blobs/uploads/ 2>&1 | grep Location:
curl -v -XPUT 'http://localhost:5000/v2/cache/blobs/uploads/<uuid from above>?_state=<state from above>&digest=sha256%3A4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1' --upload-file .registry-cache/docker/registry/v2/blobs/sha256/4f/4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1/data

the previous HEAD request returned a 200 OK response.

Based on the above, it looks like we're just failing to upload that particular blob. Looking further.