GitpodWorkspacesStuckOnStopping alert is misleading

[csweichel]

Bug description

The GitpodWorkspaceStuckOnStopping alert is misleading. It's checking if the number of workspaces in "STOPPING" is consistently higher than some (low) threshold.

It does NOT however check if it's the very same workspaces that have been in stopping for a given time.

Steps to reproduce

Stop many workspaces at the same time

Expected behavior

The alert should only fire if the same workspaces are actually stuck in stopping.

Related

xref: the runbook to improve

[jenting]

The alert should only fire if the same workspaces are actually stuck in stopping.

We could check the current time - the time of DeletionTimestamp > the workspace timeout time.

For example, if the workspace default timeout time is 1 hour, the alert could check whether the current time - the time of DeletionTimestamp > 1 hour, then trigger an alert.

[kylos101]

Closing in favor of waiting to leverage mk2, so that we can track time spent by phase, using objects defined by CRD.

cc: @aledbf @sagor999 @Furisto