High memory & CPU usage after running for awhile #178
Comments
|
Related to microsoft/vscode-go#853 So the issue is our language server resource usage is currently tuned for a server environment were we have a premise of lots of RAM. This has been presenting us issues in prod as well, so rewriting our hover and j2d to be smarter with resources needs to be done. It is almost certainly related to using the simple go loader interface and/or naive caching of its output. Add in our conservative cache invalidation policy, and we create a lot of garbage and re-work things out often in an editor environment. I'll see if there is any short term wins here, but we probably need to make some larger changes to get good at this. |
|
Same issue for me: even with 16GB of memory I had to stop using go-langserver in vscode because it was frequently using all memory, making the computer unusable and triggering the OOM-Killer... |
|
Since I can reliably cause the langserver to use all memory on my private code base, how can I help debug this issue (or at least find where most of the ressource are spent) ? Any profiling tool to run ? If so, how ? |
|
@keegancsmith Any updates on this? I have pushed a change for adding the ability to pass Is there anything else you'd like to see users able to do via vscode? |
|
I hate to do this but any news on this? this is kind of a major issue. I have 32gb of ram and the oom killer triggered twice in the last 4 hours, gocode is broken on go git. |
|
@OneOfOne Apologies for the delay. We're still looking into it, but have had to prioritize a few other features for Go in the past 2 weeks. I'll update this issue when we have a patch targeting this issue. Appreciate the feedback and thanks for trying it out! |
|
+1 Same issue here (on MacOS). Seems to grow to consume all available memory and then keeps going consuming swap as the system slows to a crawl. At one point "go-langserver" was consuming over 18GB. FYI - after restarted "go-langserver", it re-started consuming 183.6MB of space. Added a single space to a .go file and saved - memory consumed went to 576.0MB. Changed a letter from lower to uppercase, saved, changed it back to lowercase, saved and memory was at 1.09GB. Will try restarting next... OK, restarted and getting same behavior. 193.8MB immediately, added a single space character, saved - memory went to 574.8MB. Changed a letter from uppercase to lower, saved, changed it back to uppercase, saved and memory is at 1.86GB. Did it again and memory is 2.0GB. |
|
Thanks for the feedback @mgleason. I'll be targeting the local machine use case in the next couple of weeks, so I hope to have a patch out soon. In the meantime, thanks for trying it out and posting the feedback! |
|
A detailed comment for someone who tackles this problem. TLDR: Rewrite everything in loader.go so we work more like the go compiler. https://sourcegraph.com/github.com/sourcegraph/go-langserver@ee6a3562/-/blob/langserver/loader.go Currently this language server is optimized for the server environment on sourcegraph.com. The tradeoffs which most negatively effect desktop users are:
When we receive a request for Our caching for this is only to cache the output of loader for With that in mind the major improvements for this are:
Implementing this sort of stuff almost certainly will require forking and heavily modifying Note: I have not done any proper profiling of the desktop use case, this is what my educated guess says the problem is. We may be surprised to find that its something silly like |
|
(Hi all! I'm taking over here where @keegancsmith had to leave off) I've merged #192 just now which will significantly help lower the memory consumption of the language server, by somewhere between 20-40% from my testing. I suspect my above PR entirely fixes this issue, but I also have another change brewing that will lower the memory consumption even further, and make most hover/definition requests faster. I'll keep everyone here posted, I hope to have that change out by the end of this week or early next week (obviously this is just an estimate / timeline could change). Feel free to give the language server another shot and provide feedback! You can update it by running |
|
I'll test and get back to you, woo |
|
It seems stable around 4.3G of RAM (16.9G virtual), been running for 30 mins or so. |
|
It randomly happened again, it spiked to 25G and I had to ssh in from my phone to kill it. :( @slimsag |
|
Thanks for giving it a shot and providing feedback, @OneOfOne, we appreciate it! Any details you can provide would be helpful:
As mentioned in my prior message, I am working on some further improvements that will make a larger impact here. I'll update here as soon as that lands and can be tested :) |
|
Glad to help, when it works it's really responsive and great.
|
|
A simular issue is still happening on my Mac (tried updating after these changes) on a small sized project (< 30k LOC). But here is more high CPU usage than memory. After the update I could use the langserver for more time, but after a while it still happens. |
|
I'm also having the same problem here on Ubuntu. High memory and CPU usage: |
|
Hi @cossay Your screenshot shows processes like |
|
@slimsag, you are right. I have "go.useLanguageServer" set to "false" in my VS Code settings. I'm going to set it as you suggested and see what happens. I'm not working on any projects now, I'm just following a tutorial. |
|
It'd also be very helpful for me if anyone observing high memory or CPU usage could upload an SVG profile, see https://github.com/sourcegraph/go-langserver#profiling |
|
Now with almost 100% usage after ~45min: |
…rging
Typechecking used to cause linear memory growth, now it doesn't. In the docker repository, editing and hovering 5 times (to trigger typechecking) produces:
Without this change:
1. 719MB
2. 1.21GB
3. 1.89GB
4. 2.24GB
5. 2.51GB
With this change:
1. 655MB
2. 776MB
3. 787MB
4. 787MB
5. 787MB
i.e. we used to grow by about 500MB per typecheck operation, now we just idle at ~500MB.
I actually planned to remove the cache _entirely_, but with a change in direction about how I will be tackling the slowness of go to definition and hover, removing the cache entirely is not possible anymore. This is the next best thing, and is still a HUGE improvement.
Each `LangHandler` has it's own cache (see `(*LangHandler).resetCaches`), but they are backed by the same LRU cache (see `cache.go`). Every cache that wraps the LRU cache (see `newTypecheckCache`) uses a global cache ID to avoid conflicts:
```
// cacheID is used to prevent key conflicts between different
// LangHandlers in the same process.
cacheID int64
```
That is, we use the same backing LRU cache but keep our data separate. The intent of this, I presume, is to keep memory usage lower (because 2 LRU caches would have 2x the number of objects in caches overall). At first glance, this seems like a good idea. But when it comes to purging the cache, issues crop up. For example:
```
func (c *boundedCache) Purge() {
// c.c is a process level cache. Since c.id is part of the cache keys,
// we can just change its values to make it seem like we have purged
// the cache.
c.mu.Lock()
c.id = nextCacheID()
c.mu.Unlock()
}
```
This code assumes that we can purge the cache by incrementing the cache ID. And that's true -- data from the old cache ID will never be returned. However, this doesn't actually purge the old cache contents from memory!
In the case of sourcegraph.com, not purging things from memory is OK (we expect our process to hold all of it's memory, basically). But for local editors using the language server, this isn't OK. Every time the user modifies a file, the cache is purged (because the typechecking data is invalid). Because the purging doesn't actually clear contents from memory, we store (by default) up to 500 typechecked copies of the user's program. For something like docker, these cache objects can be up to 500MB of memory each (and we'll store 500 of them).
Given this is the case, we should really purge the cache contents instead of just invalidating them via ID incrementing. This will actually help lower memory consumption on sourcegraph.com, too, so it's a win-win situation.
Helps microsoft/vscode-go#853
Helps #178
…rging
Typechecking used to cause linear memory growth, now it doesn't. In the docker repository, editing and hovering 5 times (to trigger typechecking) produces:
Without this change:
1. 719MB
2. 1.21GB
3. 1.89GB
4. 2.24GB
5. 2.51GB
With this change:
1. 655MB
2. 776MB
3. 787MB
4. 787MB
5. 787MB
i.e. we used to grow by about 500MB per typecheck operation, now we just idle at ~500MB.
I actually planned to remove the cache _entirely_, but with a change in direction about how I will be tackling the slowness of go to definition and hover, removing the cache entirely is not possible anymore. This is the next best thing, and is still a HUGE improvement.
Each `LangHandler` has it's own cache (see `(*LangHandler).resetCaches`), but they are backed by the same LRU cache (see `cache.go`). Every cache that wraps the LRU cache (see `newTypecheckCache`) uses a global cache ID to avoid conflicts:
```
// cacheID is used to prevent key conflicts between different
// LangHandlers in the same process.
cacheID int64
```
That is, we use the same backing LRU cache but keep our data separate. The intent of this, I presume, is to keep memory usage lower (because 2 LRU caches would have 2x the number of objects in caches overall). At first glance, this seems like a good idea. But when it comes to purging the cache, issues crop up. For example:
```
func (c *boundedCache) Purge() {
// c.c is a process level cache. Since c.id is part of the cache keys,
// we can just change its values to make it seem like we have purged
// the cache.
c.mu.Lock()
c.id = nextCacheID()
c.mu.Unlock()
}
```
This code assumes that we can purge the cache by incrementing the cache ID. And that's true -- data from the old cache ID will never be returned. However, this doesn't actually purge the old cache contents from memory!
In the case of sourcegraph.com, not purging things from memory is OK (we expect our process to hold all of it's memory, basically). But for local editors using the language server, this isn't OK. Every time the user modifies a file, the cache is purged (because the typechecking data is invalid). Because the purging doesn't actually clear contents from memory, we store (by default) up to 500 typechecked copies of the user's program. For something like docker, these cache objects can be up to 500MB of memory each (and we'll store 500 of them).
Given this is the case, we should really purge the cache contents instead of just invalidating them via ID incrementing. This will actually help lower memory consumption on sourcegraph.com, too, so it's a win-win situation.
Helps microsoft/vscode-go#853
Helps #178
emidoots
changed the title
|
@henvic @OneOfOne (/cc @ramya-rao-a) I've made another significant improvement to the memory consumption issue. I believe this should solve the issue you've all observed with the memory usage growing constantly. Now, it should sit at some constant number based on how large your project is (e.g. it should idle at a few hundred MB instead of growing into multiple GB). The CPU usage issue / slowness around responding to go to definition and hover requests is still known, I'm working on improving this now. Please feel free to give the language server another shot and provide feedback:
|
|
I'll give it a try and report back. |
|
Good news / bad news kinda thing, good news is it stabilized at 1.6gb (~5%) of ram, bad news is every time I type anything it spikes to using 800% CPU. @slimsag |
|
Thanks for the feedback, @OneOfOne ! That is actually just good news. I am already aware of the CPU usage issue and am working on it now :) will update soon. |
This change adds a godef-based hover backend, similar to the godef-based `textDocument/definition` backend added prior. The motivation for this change is to avoid typechecking the entire program just to serve a single `textDocument/hover` request. After this change, hover and definition are both very quick and use little resources. Larger requests like `workspace/symbol`, or `textDocument/references`, will continue to use more resources as they must perform typechecking. The output style of this hover implementation does vary from our prior typechecking implementation in slight ways, but overall the implementation always produces results that are on par or slightly better than our typechecking implementation. As with the `textDocument/definition` implementation, we should attempt consolidation of this hover implementation with our typechecking variant further in the future. At this point, of course, they are too different to share much code or implementation. Helps #178 Helps microsoft/vscode-go#853
|
(Copying the message I just posted in microsoft/vscode-go#853 (comment) since these were basically the same issue.) Hi everyone ( /cc @ramya-rao-a ), this issue should be fixed now. 😃 Very significant improvements have been made to both goto definition and hover requests, and they now use a tiny fraction of the memory / CPU usage seen before. To update, please run: And once again set Please feel free to provide any feedback here, and if all looks good we can close this issue out! |
|
@slimsag thanks for working on that! I know that Go 1.9 is still in beta, but could you add support for it? Installing go-langserver on go 1.9beta1 fails: |
|
@slimsag awesome, it works! Thank you very much! :D |
|
@OneOfOne @henvic @vanackere @cossay Can you update the language server |
|
3 hours so far and it's steady at 1.5g virt mem and 0.7-1.5% cpu! |
|
Sadly after a while the cpu usage spikes to 40% and stays there even when idle. |
|
Hi @OneOfOne -- please do the following to give me some more insight (the CPU profile, unfortunately, does not provide enough here):
|
|
Ok I can't reproduce it anymore, so it seems to be working good, if I run into it I'll update this issue. Thank you for your hard work @slimsag. |
|
Alright :) Please don't hesitate to open a new issue if you run into it again (even if you have little or no info)! |
|
Closing since this issue is stale / no further complaints about performance have been made. If anyone runs into issues, please do not hesitate to create a new issue! :) |
Enabled the langserver and did a few hovers for testing. Then vscode was in the background. After a few minutes the langserver was using 5.6GB of memory (machine only has 4 GB RAM).
The text was updated successfully, but these errors were encountered: