Opening old workspaces stuck on pulling container image

[adg25]

Bug description

Recently (last 2 weeks, maybe more) members of my team have not been able to open previously closed workspaces. Opening new pods with the gitlab browser extension works just fine, but re-opening pods gets stuck on the "Pulling container image" step. I don't think there's anything out of the ordinary with our setup, we have a .gitpod.yml which specifies pulling the latest image from our docker registry hosted in gitlab. Some members of the team don't have issues with this, some do, and some are on and off.

Steps to reproduce

Open a new gitpod workspace from a gitlab merge request, close the workspace and try to open it again. Gets hung on the "Pulling container image step"

Workspace affected

All workspaces

Expected behavior

I expect the workspace to pull the container image and start properly

Example repository

.gitpod.yml file content for image:
image: registry.gitlab.com/<my_company>/path/to/hosted_images:latest

Anything else?

No response

[axonasif]

@adg25 I think it would be helpful for our workspace-team if you can share the affected workspace-id too!

[adg25]

In classic bug ticket fashion now all of my workspaces are unaffected by this issue! But another member of my team is experiencing it, his workspace is https://belvederetradin-tribe-y684d5w18ri.ws-us31.gitpod.io/

Edit: 1:14pm it's currently happening to https://belvederetradin-tribe-tke67qfj000.ws-us31.gitpod.io/

[adg25]

I tried opening a workspace I was editing yesterday and I got this error message:

Timed Out
last backup failed: cannot delete workspace from store.

Workspace is https://belvederetradin-tribe-3rrw2k0grv6.ws-us32.gitpod.io/

[adg25]

@axonasif Any update on this?

[pawlean]

Might be one for @kylos101 - copying him in.

[adg25]

Thanks! We may have some idea of what's going on here actually. We are using Bazel as the build system for our mono repo and modified the bazelrc so that

startup --output_base /workspace/bazel-cache

Which means that the bazel build cache lives in /workspace instead of the default /home directory. We did this because the cache would take up more than 5Gi of space and crash all of our workspaces, and a recommendation from the gitpod discord was to utilize the 30Gb we get from /workspace. However we think that when workspaces stop with a built cache the images are too large to be pulled again (and subsequently time-out. I've confirmed that if I execute a bazel clean before stopping a workspace then it is able to open just fine. Here's some output from df before and after a bazel clean

gitpod /workspace/monorepo $ df -h /workspace
Filesystem      Size  Used Avail Use% Mounted on
/dev/md42        30G   12G   19G  37% /workspace
gitpod /workspace/monorepo $ bazel clean
(22:17:58) INFO: Invocation ID: 26b2127d-e11d-4550-a638-283a5971fa03
(22:17:58) INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
gitpod /workspace/tribe $ df -h /workspace
Filesystem      Size  Used Avail Use% Mounted on
/dev/md42        30G  3.5G   27G  12% /workspace
gitpod /workspace/monorepo $

[kylos101]

Thanks for the heads up @adg25 !

How large of a /workspace were you having trouble with?

[adg25]

@kylos101 Certainly the 12G there is a problem to re-pull the container image after it's stopped - and the 3.5G is good enough (less than a minute usually). I don't really have a metric for "at X size we start having these issues", but there's rarely an in-between for devs anyway since usually we either operate with a full bazel cache or an empty one.

[kylos101]

Thanks for the heads up, @adg25 ! I've scheduled this for our team to do some research. Have a nice weekend! Assuming we can recreate the problem, that should give us some better insight.

While I'm thinking about it, if you're able, could you share a simple repo where you're able to reproduce the problem? It's okay to say no, just figured I would ask.

[adg25]

Thanks @kylos101. Unfortunately I can't share the repo with you since it's private but I can share some relevant metrics. We currently have 165 bazel BUILD files that include java_* cc_* py_* and container_* (library/binary/image/push) modules and in total about 47k lines of code

[kylos101]

Thanks, @adg25 that should help us recreate a similarly sized /workspace.