Improve monitoring and feedback for errors during workspace cluster selection

[kylos101]

Bug description

On start, some workspaces get stuck on "Allocating resources (from Discord)".

Another example from Twitter.

Steps to reproduce

Unconfirmed, but, try to start a workspace when prebuild load is very high.

Workspace affected

vipertools-css-eamifbfe80m

Expected behavior

It should start, instead of getting stuck.

Example repository

n/a

Anything else?

The issue occurred for some users on January 25 @ 10:52pm UTC.

[thisisommore]

I can confirm this, in my case every first workspace of the same context is stuck

[kylos101]

This may be an issue with server and the # of socket connections, and live on the WebApp side of the house.

To start, let's gauge the frequency for this error by searching in the logs. Do the entries for the error line up with peeks in our volume? If yes, a simple mitigation may be to scale up server (+1 replica).

This may be difficult to recreate, but please try, and report the result here.

cc: @jldec @JanKoehnlein as a heads up. I am going to add this to your inbox as well, to get your thoughts.

[JanKoehnlein]

Thanks for the heads up. Let us know if you have a bit more information.

[thisisommore]

I can confirm this, in my case every first workspace of the same context is stuck

Now it is working I guess, never got into it again