Starting workspace from prebuild -- permission denied, last backup failed, workspace does not exist

[chuck-confluent]

Bug description

Error when pulling prebuild:

cannot initialize workspace: cannot initialize workspace:
prebuild initializer:
prebuild initializer:
unlinkat /dst/.docker-root/overlay2/05b88337aa8c77ea82e4150b09f65ae9fe390cdaf53861a4f8e2db9f368c9aef/diff/usr/bin/confluent-hub:
permission denied; last backup failed: workspace does not exist.

Steps to reproduce

Launch https://gitpod.io/#https://github.com/confluentinc/cp-demo

Workspace affected

confluentinc-cpdemo-q8r5zrnkeds

Expected behavior

No response

Example repository

https://github.com/confluentinc/cp-demo commit ebffce1

Anything else?

It looks like the error has to do with an executable confluent-hub inside a docker image that was baked into the prebuild. Really confused on this one.

[pawlean]

Hi @chuck-confluent!

You may be seeing this due to our recent incident. It's worth keeping an eye out on it.

[chuck-confluent]

@pawlean is there a workaround to run the repo without a prebuild?

[chuck-confluent]

update: Launching a workspace for this repo no longer attempts to pull down the prebuild snapshot. It launches from the base workspace. That's good, I suppose. I look forward to prebuilds working again.

Another observation -- I notice running a manual prebuild with gitpod.io/#prebuild/ runs a prebuild but it doesn't show up in my gitpod.io/projects//prebuilds and workspaces don't seem to be pulling from it

[sagor999]

Customer reported same issue in front (internal): https://app.frontapp.com/inboxes/teammates/9932874/inbox/open/41825492426

Looking in logs (internal): https://cloudlogging.app.goo.gl/rwv9Ld3m8aTntM6V6
there seems to be a lot of those errors about getting permission denied when accessing /dst/.docker-root/overlay2

@utam0k @Furisto would you guys be able to take a look at it (when you have a chance)
I am not quite sure why there is permission denied there and what exactly it is attempting to do.
@kylos101 for 👀

For customer from Front link:
This seems to do with running docker-compose pull and docker-compose build in init task. If that is moved out of init part, then prebuild works just fine. 🤔

[utam0k]

hmm... My prebuild is success with https://github.com/confluentinc/cp-demo on SaaS

[chuck-confluent]

I’ve been having successful prebuilds for a little while now too. Feel free to close this issue unless it’s helpful to keep investigating. It maybe had something to do with the big outage that happened in April?

[sagor999]

Created a new issue so that we don't tangle this two together:
#10108