[ws-manager] potential go routines leak

[jenting]

Bug description

In us47, the ws-manager seems to have potential go routines leak, and it doesn't happen in eu47. 🤔

The symptom is that the ws-manager handles the gRPC request (such as creating workspace) pretty slowly. So, the user experience is they can't open the workspace.

As a workaround, we restart the ws-manager, after that, the go routines back to normal.

Steps to reproduce

No idea how to reproduce it so far.
And here is the internal slack thread.

Workspace affected

No response

Expected behavior

No response

Example repository

No response

Anything else?

No response

[NotNoah]

I've noticed in my Gitpod workspaces very recently (starting last week) that once I connect to my desktop VS code the memory from gp top increases 3-4gb and keeps climbing until crash, but it doesn't do this if i only use the web version of VS Code. Could it be related?

This happens to me using Gitpod SaaS and also on the June release of Gitpod Self-Hosted

[jenting]

From the pprof I dumped, the Goroutine leaks seem to happen in http(*persistConn)readLoop and http(*persistConn)writeLoop.

[jenting]

We still see the Goroutines increase in the us54 cluster. Reopen it.

[kylos101]

eu54 does not appear to have the issue:

here is us54:

perhaps the issue is related to prebuilds, which generally occur in US region.