High memory & CPU usage after running for awhile

[beyang]

Enabled the langserver and did a few hovers for testing. Then vscode was in the background. After a few minutes the langserver was using 5.6GB of memory (machine only has 4 GB RAM).

[keegancsmith]

Related to microsoft/vscode-go#853

So the issue is our language server resource usage is currently tuned for a server environment were we have a premise of lots of RAM. This has been presenting us issues in prod as well, so rewriting our hover and j2d to be smarter with resources needs to be done. It is almost certainly related to using the simple go loader interface and/or naive caching of its output. Add in our conservative cache invalidation policy, and we create a lot of garbage and re-work things out often in an editor environment.

I'll see if there is any short term wins here, but we probably need to make some larger changes to get good at this.

[vanackere]

Same issue for me: even with 16GB of memory I had to stop using go-langserver in vscode because it was frequently using all memory, making the computer unusable and triggering the OOM-Killer...

[vanackere]

Since I can reliably cause the langserver to use all memory on my private code base, how can I help debug this issue (or at least find where most of the ressource are spent) ? Any profiling tool to run ? If so, how ?

[ramya-rao-a]

@keegancsmith Any updates on this?

I have pushed a change for adding the ability to pass -trace and -logfile flags to the language server (See microsoft/vscode-go@50e5c8a)

Is there anything else you'd like to see users able to do via vscode?

[OneOfOne]

I hate to do this but any news on this? this is kind of a major issue.

I have 32gb of ram and the oom killer triggered twice in the last 4 hours, gocode is broken on go git.

[beyang]

@OneOfOne Apologies for the delay. We're still looking into it, but have had to prioritize a few other features for Go in the past 2 weeks. I'll update this issue when we have a patch targeting this issue. Appreciate the feedback and thanks for trying it out!

[mgleason]

+1 Same issue here (on MacOS). Seems to grow to consume all available memory and then keeps going consuming swap as the system slows to a crawl. At one point "go-langserver" was consuming over 18GB.

FYI - after restarted "go-langserver", it re-started consuming 183.6MB of space. Added a single space to a .go file and saved - memory consumed went to 576.0MB. Changed a letter from lower to uppercase, saved, changed it back to lowercase, saved and memory was at 1.09GB. Will try restarting next...

OK, restarted and getting same behavior. 193.8MB immediately, added a single space character, saved - memory went to 574.8MB. Changed a letter from uppercase to lower, saved, changed it back to uppercase, saved and memory is at 1.86GB. Did it again and memory is 2.0GB.

[beyang]

Thanks for the feedback @mgleason. I'll be targeting the local machine use case in the next couple of weeks, so I hope to have a patch out soon. In the meantime, thanks for trying it out and posting the feedback!

[keegancsmith]

A detailed comment for someone who tackles this problem.

TLDR: Rewrite everything in loader.go so we work more like the go compiler. https://sourcegraph.com/github.com/sourcegraph/go-langserver@ee6a3562/-/blob/langserver/loader.go

Currently this language server is optimized for the server environment on sourcegraph.com. The tradeoffs which most negatively effect desktop users are:

• Many shared read-only workspaces in a single process.
• Server environment has lots of RAM.

When we receive a request for my/pkg/a.go, we use golang.org/x/tools/go/loader to typecheck my/pkg. We only typecheck my/pkg, but the bulk of the time is spent on the transitive dependencies. https://sourcegraph.com/github.com/sourcegraph/go-langserver@ee6a3562/-/blob/langserver/loader.go#L233

Our caching for this is only to cache the output of loader for my/pkg. All the intermediate work it does (on the transitive dependencies) is not cached at all. When we receive an edit we just throw that cache out. This works alright for sourcegraph.com since when a user is browsing a file we never invalidate the cache and we expect to receive a lot of hovers for different places in that file. On the desktop when we receive an edit, we just throw away all of our caches https://sourcegraph.com/github.com/sourcegraph/go-langserver@ee6a3562/-/blob/langserver/handler.go#L353

With that in mind the major improvements for this are:

• Granular caching and cache invalidation. Do not throw away everything on edits.
• Cache intermediate work. Look into how we can use what go stores in $GOPATH/pkg.
• Possibly throw away the coarse pkg caching, and instead do the minimal amount of work for a hover, rather than doing the work for a whole pkg. This will technically make sourcegraph "slower", but I believe it will still be fast enough for users (instead of a few milliseconds, it will be a few 100 milliseconds).
• Do not use in-memory caches, instead rely on a filesystem (or make it pluggable).

Implementing this sort of stuff almost certainly will require forking and heavily modifying golang.org/x/tools/go/loader. I would target the j2d use case since that is the most heavily used. We can then get hover to use the j2d result to generate hover contents. At some point we had a fork of godef. Making godef use a virtual filesystem may be the easiest win. As for references it can be left alone since it works quite differently. Symbols just relies on the AST, so does not suffer the same performance problems.

Note: I have not done any proper profiling of the desktop use case, this is what my educated guess says the problem is. We may be surprised to find that its something silly like FindPackage being slow. If so, we could look into open sourcing the optimised FindPackage we use on sourcegraph.com.

[emidoots]

(Hi all! I'm taking over here where @keegancsmith had to leave off)

I've merged #192 just now which will significantly help lower the memory consumption of the language server, by somewhere between 20-40% from my testing.

I suspect my above PR entirely fixes this issue, but I also have another change brewing that will lower the memory consumption even further, and make most hover/definition requests faster. I'll keep everyone here posted, I hope to have that change out by the end of this week or early next week (obviously this is just an estimate / timeline could change).

Feel free to give the language server another shot and provide feedback! You can update it by running go get -u github.com/sourcegraph/go-langserver and enabling the VS Code option once again.

[OneOfOne]

I'll test and get back to you, woo