ggcr: `mutate.Append` creates invalid images when appending duplicate layers (same diff ID) but different blob

[mafredri]

Describe the bug

It's possible to mutate an image with a duplicate layer that has the same diff ID as a previous layer (from a parent image), but the underlying blob differs due to a difference in compression (e.g. level).

To Reproduce

docker volume create ggcr-test-data
docker run -d -p 1234:5000 --name ggcr-test --volume ggcr-test-data:/var/lib/registry registry:2

mkdir -p tmp/docker tmp/kaniko
echo $'FROM alpine:latest\nWORKDIR /home' >tmp/docker/Dockerfile
echo $'FROM localhost:1234/docker:latest\nRUN echo test' >tmp/kaniko/Dockerfile

docker build -t localhost:1234/docker:latest tmp/docker
docker push localhost:1234/docker:latest

docker run \
    --network=host \
    -v $(pwd)/tmp/kaniko:/workspace \
    gcr.io/kaniko-project/executor:latest \
    --dockerfile /workspace/Dockerfile \
    --destination "localhost:1234/kaniko:latest" \
    --context dir:///workspace/ \
    --force-build-metadata

Output (end):

INFO[0000] Pushing image to localhost:1234/kaniko:latest
error pushing image: failed to push to destination localhost:1234/kaniko:latest: PUT http://localhost:1234/v2/kaniko/manifests/latest: MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1

Thus we see that 4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 from the Docker build is missing and replaced with the one from Kaniko (both the empty layer, different compression result). This is a manifest<->layer inconsistency.

Expected behavior

The mutation does one of the following:

• Looks through image layers for a matching diff ID and replaces the given layer with it
• Returns an error due to matching diff ID but otherwise mismatched digest/blob

Or perhaps something else should happen, I haven't researched the subject in-depth.

Additional context

This can easily be worked around outside the library by using LayerByDiffID(...) before calling mutate.Append as done here: coder/kaniko@131d609. But I don't think it's a good idea to allow mutate to create invalid images.

• Version of the module: v0.20.2
• Registry used (e.g., GCR, ECR, Quay): registry:2

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Keep fresh with the 'lifecycle/frozen' label.

[mafredri]

Bump

[JacobHenner]

I believe I'm encountering the same issue.

We have some tooling that extracts tarballs from RPMs and appends them as container layers to existing images using crane append.

If someone builds image_A, appends $package (following the procedure above), and then someone else builds image_B using image_A as its parent, and then also (unnecessarily) appends $package (following the procedure above), the resulting image cannot be pushed by Kaniko. The same MANIFEST_BLOB_UNKNOWN: blob unknown to registry error message is displayed, failing on the digest of the duplicated layer.

Perhaps the fix from coder/kaniko#35 should be incorporated upstream? I understand that Kaniko is a separate project from go-containerregistry. I'm not immediately sure whether the limitation is actually a bug in Kaniko, or an error in how go-containerregistry deals with duplicate diff ID layers.