[image-builder-mk3] Does not carry over env vars from base images

[geropl]

The underlying issues seems to be that env variables from base iamges are not carried over to the resulting image.
For instance, workspaces on gitpod-io/gitpod could not find gcloud because it was not in the PATH (but in the image).
Switching to old image-builder did fix this.

Incident: https://www.gitpodstatus.com/incidents/fk6prflgvvth

User reports:

• [image-builder] workspace images' environment variables are broken #5594
• Dependencies lost upon workspace reopening #5589

[meysholdt]

we can re-enable image builder mk3 on staging, in case that is helpful: https://github.com/gitpod-com/gitpod/pull/5901 Thx @wulfthimm for the PR.

[geropl]

An integration test for this: #5609

[geropl]

Note that work on this on devstaging will likely be blocked by this PR: #5497

[csweichel]

/schedule

[roboquat]

@csweichel: Issue scheduled in the workspace team (WIP: 0)

In response to this:

/schedule

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[iQQBot]

I think because mk3 use buildkit LLB

gitpod/components/image-builder-bob/pkg/builder/builder.go

Line 200 in a4a8f39

def, err := llb.Image(b.Config.BaseRef).Marshal(context.Background())

Low-level builder only works on snapshots. There are no methods for controlling image metadata changes. Image metadata can be managed by a frontend, should it be needed. The only component that could know about the image format is image exporter.

[csweichel]

I think because mk3 use buildkit LLB

That's indeed the problem. llb.Image does nothing with the source image's metadata if there's no imagemetadataresolver present. Stepping through the code shows that if that's present the metadata is downloaded and applied to the state**. That said, the environment variables are still not present, and while debugging I've not seen the addenv state functions actually applied. I reckon we're still using llb wrong here somehow.

** the default imagemetadataresolver doesn't care much about authentication, this will be a problem

[csweichel]

I've filed an issue with buildkit: moby/buildkit#2362

[csweichel]

/assign