ws-daemon go routine leak

[ArthurSens]

Describe the bug

ws-daemon has go routine leaks, as shown by our monitoring data.

Go Routines:

Even though it looks strange that one pod is increasing a lot faster than the others, all pods are slowly (but steadily) increasing the number of go routines.

[aledbf]

@ArthurSens do you know how to trigger the leak?

[ArthurSens]

Sorry @aledbf, I don't have that information. We still need to investigate 🙁

[aledbf]

@ArthurSens I profiled the ws-daemon component in production and find out this is not a leak. Each workspace creates a goroutine for the daemon.sock socket file.

[ArthurSens]

I'm not sure I understand how it explains it isn't a leak 🤔. The amount of goroutines is clearly increasing over time and will only go down after a restart. As you can see on the issue description, goroutines start at about a hundred and are only increasing as time passes. Reaching almost 500 avg in a week.

I just took another look at this same dashboard, and things are looking more troublesome after today's deployment!

A restart happened, so all leaked go routines dropped, but the increase rate is a lot higher now:

[ArthurSens]

Maybe this graph from the EU cluster shows the difference better:

Amount of go routines are going to the roof after today's deployment