Image id shown as missing with docker history

[daveisfera]

The layers generated by buildkit don't play nice with the docker history command and show <missing> instead of the layer id for IMAGE

[tonistiigi]

The field in the history output is parent image ID, not layer ID. There is no parent image, therefore the field is not shown. Same as for pulled images for example.

[daveisfera]

Sorry, I used the wrong term, but shouldn't the ID be showing instead of <missing> when building a layer using buildkit?

[tonistiigi]

There is no such thing as layer ID in Docker API. There is only diffid in rootfs that you can see if you run inspect on an image. The field you are looking at is a reference to a parent image and buildkit does not create any side images as part of the build.

[daveisfera]

It creates layers that are used for the cache and I'm expecting to be able to see those so I can verify which layers are being used for caching and that they're shared across different images correctly. Is there a way to look at that info with buildkit other than docker history?

[tonistiigi]

Is there a way to look at that info with buildkit other than docker history?

I don't understand how you could be using history for this. Eg. docker pull redis && docker history redis will give you exactly the same output. As I said, there is no layer type in Docker API, so no special API for doing this. Closest is the rootfs object you see in image inspect.

[daveisfera]

I'm not doing a docker pull. I'm doing a docker build myimage1 and docker build myimage2 that have the first few layers with the same commands because it's common code that's used by both of those images/services. Then I run a docker history on both to verify that the layers that should be shared are shared.

[torson]

I used to do the same thing to optimize the image layers of the whole app stack - 10 images reusing as many layers as possible.
Now with Buildkit that is not possible anymore using the docker image history command.
But you can use another command for getting the layer sizes :

docker manifest inspect IMAGE:TAG
docker manifest inspect IMAGE:TAG | jq '[.layers[]]'

Check this answer: https://stackoverflow.com/questions/40377199/how-to-get-total-size-of-a-docker-image-by-docker-api-properly

[tonistiigi]

Building with buildkit does not hide the layer sizes from the history array.

[tonistiigi]

Then I run a docker history on both to verify that the layers that should be shared are shared.

To check if the layers are shared you can run docker inspect <id> | jq '.[].RootFS.Layers'

[torson]

Sorry, forgot to mention extra detail. So what I need is a list of hashes together with their sizes , this enables me to get the full size of the app as the sum of all unique layers. docker manifest inspect gives me that list . docker history has sizes only , and docker inspect has hashes only .
Maybe there's another command in addition to docker manifest inspect that would work on local images.
For me it works only on images that are pushed to a registry, for local images I get

errors:
denied: requested access to the resource is denied
unauthorized: authentication required

One can of course spin up a local registry though that doubles the storage size of those images.

[asottile]

here's the concrete difference for me -- but boiled down to a very minimal example:

FROM ubuntu:jammy
RUN echo hi > f
RUN echo hello > f2

DOCKER_BUILDKIT=0 docker build -t img .
DOCKER_BUILDKIT=1 docker build -t img-buildkit .
docker image history img --format='{{.ID}}\t{{.CreatedBy}}'
docker image history img-buildkit --format='{{.ID}}\t{{.CreatedBy}}'

and the outputs of the history commands since those are the only ones relevant to me:

$ docker image history img --format='{{.ID}}\t{{.CreatedBy}}'
da7370b80ece	/bin/sh -c echo hello > f2
3de4f0655aa2	/bin/sh -c echo hi > f
74f2314a03de	/bin/sh -c #(nop)  CMD ["/bin/bash"]
<missing>	/bin/sh -c #(nop) ADD file:fb4c8244f4468cdd3…
<missing>	/bin/sh -c #(nop)  LABEL org.opencontainers.…
<missing>	/bin/sh -c #(nop)  LABEL org.opencontainers.…
<missing>	/bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH
<missing>	/bin/sh -c #(nop)  ARG RELEASE

$ docker image history img-buildkit --format='{{.ID}}\t{{.CreatedBy}}'
659b69d1971f	RUN /bin/sh -c echo hello > f2 # buildkit
<missing>	RUN /bin/sh -c echo hi > f # buildkit
<missing>	/bin/sh -c #(nop)  CMD ["/bin/bash"]
<missing>	/bin/sh -c #(nop) ADD file:fb4c8244f4468cdd3…
<missing>	/bin/sh -c #(nop)  LABEL org.opencontainers.…
<missing>	/bin/sh -c #(nop)  LABEL org.opencontainers.…
<missing>	/bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH
<missing>	/bin/sh -c #(nop)  ARG RELEASE

with buildkit disabled I can run individual layers of the built image (for example to inspect partially built state, etc.) but I cannot get those image ids with buildkit:

$ docker run --rm -ti 3de4f0655aa2 bash
root@ea2638b1687b:/# cat f
hi
root@ea2638b1687b:/# cat f2
cat: f2: No such file or directory

I can hack around this by converting the image to a multi stage build -- but that's a lot of unfortunate complexity:

FROM ubuntu:jammy as minimal

RUN echo hi > f

FROM minimal as final
RUN echo hello > f2

DOCKER_BUILDKIT=1 docker build --target minimal -t img-buildkit-multi-minimal .
DOCKER_BUILDKIT=1 docker build --target final -t img-buildkit-multi .