Create an alert for failing image-builds (in app clusters)

[geropl]

There are several ways we could do this. The most sustainable from our perspective would be if we could have a metric for the failure rate of all image builds we trigger from server. This way it remains operational even after we move image-builds into workspace clusters.

[svenefftinge]

Would it make sense to create an alert for failed workspace starts instead? This would also catch cases where e.g. no ws-manager is reachable, i.e. any failure that would happen on webapp side.

[geropl]

@svenefftinge A quality of an alert is that it's precise, and you know how to act when it rings. If we would cover the "failed workspace rate" (which I understood you're supposing...?) it would include errors that a) are already covered by the "workspace error rate" (team workspace) + b) more user errors, and we'd need to tell apart on alert.

[svenefftinge]

The current situation is that we don't have alerts for workspace starts on webapp side. I'd rather get an alert and need a minute to figure out what happened than see no alert at all.

[svenefftinge]

FWIW I would trigger it on errors before ws-manager accepted a workspace start request. So doesn't include failures in workspace.

[geropl]

FWIW I would trigger it on errors before ws-manager accepted a workspace start request.

Ok, that's more on-point I think 💯

FYI There already is a metric "failed instance start" (here) which atm only handles very specific errors. We could think about expanding that one + adding an alert.