[PVC] ws-manager event workers hang forever once over 100 VolumeSnapshots and ws-manager restart

[jenting]

Bug description

The ws-manager contains 100 workers to handle the events from the workspace pod and volume snapshot.
If there are over 100 VolumeSnapshots are ready to the ws-manager be restart, the ws-manager can't handle any pod and volume snapshot event because all the workers hang forever.

Steps to reproduce

• Create over 100 VolumeSnashots.
• Try to open the workspace, the workspace pod can't be up and with Error state, and the workspace pod log is

  {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"socket did not appear before context was canceled: errors of connect to ws-daemon","level":"error","message":"cannot connect to daemon from ring0","ring":0,"serviceContext":{"service":"workspacekit","version":"commit-8d009263e19017f630e6f12436d3aa9186b8355c"},"severity":"ERROR","time":"2022-09-13T06:33:39Z","workspaceId":"gitpodio-empty-egk6yo5d3ix"}
  {"level":"info","message":"ring0 stopped","ring":0,"serviceContext":{"service":"workspacekit","version":"commit-8d009263e19017f630e6f12436d3aa9186b8355c"},"severity":"INFO","time":"2022-09-13T06:33:39Z","workspaceId":"gitpodio-empty-egk6yo5d3ix"}

Workspace affected

No response

Expected behavior

The workspace can be started.

Example repository

No response

Anything else?

#7901

https://gitpod.slack.com/archives/C04245JPHKL/p1663223540516549