Transparently re-try image builds if one fails with error code "INTERNAL"

[geropl]

During image builds we often experience short-lived situations where we starve for resource, e.g. disk space. For this reason we designed the system to be idempotent so we can re-start with confidence.

We don't do so, however. This should be as simple as adding a re-try loop with backoff.

This would catch issues like this: https://community.gitpod.io/t/error-build-failed-when-trying-to-build-a-docker-image/4216

/cc @csweichel suggested this(/sth similar?) yesterday as well

[csweichel]

/schedule

[csweichel]

We can implement this behaviour for image-builder-mk3 easily, but mk2 makes no distinction between system failure and "user input errors".

I'd propose we take on this task once #4812 is done.

(removing from groundwork right now - please re-add once mk3 is merged)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.