[image builder] return meaningful error messages to users when an image build fails

[kylos101]

Is your feature request related to a problem? Please describe

When image builds fail now, we return a generic message and error code, such as headless task failed: exit status 1

Describe the behaviour you'd like

When a docker build fails, after we've started interpretting the Dockerfile, return the actual error to the user (like a base image that doesn't exist), rather than headless task failed with an error code.

When a docker build fails, before we've started interpretting the Dockerfile, continue to return a generic message and error code.

When a failure happen, especially for user failures, log the error they would have seen (so we can more concretely know what failed).

Additional context

#14919
https://github.com/gitpod-io/gitpod/issues?q=is%3Aissue+is%3Aopen+image+build+error+message

[kylos101]

@sagor999 for 👀 and 🧑‍🔧

[kylos101]

@akosyakov for 👀 and 💭

[sagor999]

I believe it might be working correctly as of right now:

[kylos101]

@sagor999 when failure happens due to a bug in the dockerfile, can we log a more useful error? My recollection is that our logging is poor now.

Can we log text that helps us understand the origin of the failure - like the line that failed, similar to what you shared above?

[sagor999]

@kylos101 we do log a good message. See above screenshot. it shows which line and why failed in dockerfile

[kylos101]

@sagor999 can you share a link to logs explorer where you see the above error? I suspect you will not be able to find one. This is what we return while streaming the result of the image build...we may not be logging the actual content.

[sagor999]

@kylos101 Hm. As of right now there is no way to "send" actual error message from build process into parent process, so that we can log it.

We could potentially redirect stderr into file, and then in case of error, open and read that file and append it to the error. 🤔
Over here:

gitpod/components/image-builder-bob/pkg/builder/builder.go

Lines 155 to 164 in 8da6bee

	buildctlCmd.Stderr = os.Stderr
	buildctlCmd.Stdout = os.Stdout
	env := os.Environ()
	env = append(env, "DOCKER_CONFIG=/tmp")
	buildctlCmd.Env = env
	if err := buildctlCmd.Start(); err != nil {
	return err
	}

But I am not sure if this is the best approach. 🤔

[sagor999]

@WVerlaek you worked with image builder before. Do you have an idea on how to best be able to catch the error log output from builder and pass it parent process? See above message.

[WVerlaek]

@sagor999 you could maybe do something here to extract and log the error logs from a build workspace: https://github.com/gitpod-io/gitpod/blob/main/components/image-builder-mk3/pkg/orchestrator/monitor.go#L230. Not sure which parent process you mean 🤔

[sagor999]

So we want to be able to "save" image build errors in gcp log, so that we can locate them if needed.
We kick off image build over here:

gitpod/components/image-builder-bob/pkg/builder/builder.go

Line 162 in 1715273

if err := buildctlCmd.Start(); err != nil {

It uses buildctl command to build the image. Errors from this process we want to save and return them as part of the err from that function.
The problem is that buildctl dumps all of its output into stderr. I was hoping that it would only dump errors in there, and made a test using buildctlCmd.StderrPipe() that would pipe out any data from stderr into a buffer that I can append to the error if err = buildctlCmd.Wait() failed (which happens is image build fails).
But alas buildctl dumps everything into stderr.
So that is a no go, as that output can be very long for big image builds.

[sagor999]

@atduarte you had some problems with image build in the past where it was failing, but without providing you a reason as to why. Do you know if that still happens?

[atduarte]

@sagor999 when it happened last time it was because there were system errors that were causing it to fail super frequently. Unsure how to reproduce it now.

[sagor999]

@kylos101 to summarize: I think we can close this issue, since we do return meaningful errors to users now. So seems like this can be closed, as no more work\research is needed here.

[kylos101]

@sagor999 agreed, thank you for researching and finding that issue with Git! Closing as won't fix.