always display image hashes

[TrentonAdams]

It's tough to debug docker building when I can't just get into the previously successful intermediate build image and run the next command manually...

docker run -it --rm hash_id bash
# execute the next RUN line here manually.

I would therefore argue that image hashes should always display, just like they do in the current docker.

[AkihiroSuda]

There is no image hash for an intermediate cache because it is not exported to the docker image store.

maybe we should have buildctl build --debugger image=busybox that invokes an interactive busybox shell image with bind-mount for the failed build rootfs on failure.

We can also consider having buildctl build --output type=image,name=foo,output-on-failure=true but it might require some changes on the solver design.

[TrentonAdams]

There is no image hash for an intermediate cache because it is not exported to the docker image store.

Intermediate build images are indeed cached in the local docker image store, I use them all the time to find out what happened at a particular step. Every single docker line creates a permanent cached image, and can be ran using docker run --rm --it hash sh to diagnose build issues at that step, or rather the step prior to when the error occurred. If this was not the case, builds would take a long time every time.

I will think about your other comments when I get back to work. But, one thought is to drop to a shell at a particular step, like build --debugstep=7. But, in either case, having the cached image hash would allow one to jump into any step and see what went wrong.

[TrentonAdams]

Oh, and printing that small hash on the screen every time improves "usability", in that people don't have to know some special parameter to debug build images. If it were the left most item, it would also then be in the same spot every time, the same size every time (resulting in a more columnar output), and therefore improve readability of the output as well.

[niklasbuechner]

I am also trying to debug a docker build and expected all intermediary builds to be exported as an image, since that was the previous behaviour.

@AkihiroSuda Is there currently a way to look into the filesystem after the last successful build step?

[TrentonAdams]

@niklasbuechner I don't expect build kit is changing the behaviour of the building of intermediate cached images. I expect it's just not displaying the hash ids. What I don't know is if there's a way of listing them after the fact, for workaround purposes.

[dko-slapdash]

Yeah, setting DOCKER_BUILDKIT=1 disimproves debug experience a lot. Without DOCKER_BUILDKIT, it was possible to run a container with a layer right before the failed step, and with DOCKER_BUILDKIT it's not possible anymore since it just doesn't show image cached IDs.

This looks like a clear regression...

[dko-slapdash]

@AkihiroSuda, could you please advise us, is there anything changes regarding debugging of failed builds while DOCKER_BUILDKIT=1?

How do you debug such cases in your own practice?

I.e. if the build fails, I'd like to somehow spawn the container with the layer before the failed step and bash there. Previously it was possible with docker run -it --rm hash_id bash, but now DOCKER_BUILDKIT=1 doesn't show the layer hash...

[AkihiroSuda]

My own (bad) practice is RUN sleep infinity and nsenter

[TrentonAdams]

@dko-slapdash I do not think you can debug them. It's a serious failing of the build kit feature.

[dko-slapdash]

My own (bad) practice is RUN sleep infinity and

@AkihiroSuda is there some alternative approach or there is none? If not, how much is it painful in practice, are you satisfied with it in general or not quite? (Just curious.)

[AkihiroSuda]

Not satisfied, but can't come up with other approach that works with existing releases

[silviot]

My own (bad) practice is RUN sleep infinity and nsenter

This simple pointer saved my day. I knew about nsenter but forgot about it. I recommend to try this technique to anyone stuck that came here for help. True, it's not sexy. But it works.

[nottrobin]

I don't understand this technique - I added RUN sleep infinity, but how do I use nsenter to find the running image?

[AkihiroSuda]

sudo nsenter -a -t $PID_OF_SLEEP sh