Automate image-builder clean up (issue from internal gitpod-com repo)

[corneliusludmann]

Better prevent full disks of the image builder.

For transparency reasons, moving the internal issue to the public repo.

See also internal issue: https://github.com/gitpod-com/observability/issues/176

[svenefftinge]

Image Builder MkIII would solve this too I think.
@csweichel what is the current timeline for that?

[csweichel]

This is issue is definitely a nuisance and impacts our on-call effort - it's not directly user impacting though.
As indeed mkIII would solve this issue, too, I'd rather schedule an item to wrap up mkIII.

[corneliusludmann]

@csweichel I would not agree with the statement that it “only” affects our on-call effort. At least not until we are reliably able to clean up the node disks before users are impacted. Yesterday we had an outage in the US of about 1 hour due to this issue (see internal thread). Today we had a full disk on the image builder node in EU as well. Fortunately, we were fast enough with the deletion, so at least no users complained.

Either we have to find a solution to prune the image builder manually before the hard disk is completely full (some kind of maintenance alert on 50 % disk full etc.), or we have to solve this issue as soon as possible. Otherwise, we will keep having minor outages from time to time, IMHO.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.