x/tools/gopls: high memory consumption on 19MLoC input #73709
Labels
BugReport
Issues describing a possible bug in the Go implementation.
gopls/performance
Issues related to gopls performance (CPU, memory, etc).
gopls
Issues related to the Go language server, gopls.
Tools
This label describes issues relating to any tools in the x/tools repository.
Milestone
Comments
bytemain
added
gopls
ProfileTypes of profiles available:
allocsThe size of the allocs.log is 170M. just paste some code of it: full goroutine stack dump |
gabyhelp
added
the
BugReport
|
Thanks for the bug report, and sorry for the terrible experience. There's a lot to digest here. My first thought was that this was due to the set of new analyzers we recently added, but I see you are using gopls 18, so that can't explain it. The backtraces show clearly that type checking is the bottleneck, and type checking was fused with analysis in CL 622038; that optimization was included in the 18 release, so it should be in your favor. We will investigate. |
adonovan
added
the
gopls/performance
|
If you need more information, please let me know. I thought it may related to the dependency I use, because many project has this momery problem in bytedance |
|
It seems that I figured out why it consuming so much memory. Our project is using https://github.com/cloudwego/kitex as RPC framework, it will generate code from the I used binary search to find the dependency causing the memory issue, and download them in a dedicate GOPATH. When I remove those dependencies, the memory consumption is down to ~800M:
|
|
Well down for tracking it down. Hot take: any code generator that--for whatever reason--emits 19 million lines of code for a project of modest size (you reported 15K files, so I estimate a only few million lines total) is broken by design. File an upstream bug, or avoid the dependency entirely if you can. |
adonovan
changed the title
|
Thanks for you response~
and the memory usage reduced to 10.8g
|
|
and sadly, kitex is used extensively in bytedance's golang projects, so we cannot remove it. does gopls have a plan to optimize the memory usage of large codebase? goland use 4gb in the same repo and gopls will consume ~45gb |
|
or it there an setting that can let user ignore to indexing some dep, this also can help to us |
|
Unfortunately removing blank lines won't significantly reduce the memory consumption. Each blank line costs only about 1 byte (for '\n') + 1 word (in token.File.lines), plus a small amount extra because varint encodings of our various indexes will be longer. So, eliminating them will save a little more than about 9B * (16.2 million - 7.7 million) = 76MB, which is not a lot. (What difference did you observe?) The problem is not the spaces between the declarations: it's the declarations themselves. We don't have a setting to ignore large packages. Even if we did, it would dramatically worsen the user experience, as essentially all features would be disabled in any package that transitively depends on one of the suppressed large packages. A question: do you notice that the memory usage eventually settles down after an initial large (~minutes, I suspect) burst of computation? I would expect that, once computed, all the information about the large packages would be saved in the index, and would only need to be recomputed if you make changes to those packages (e.g. edit the IDL and re-run the generator). So it might be quite usable if you're patient the first time. It shouldn't take longer for gopls to index your workspace than it takes for a clean build (indeed, it should be much quicker). How long does a clean build take? If gopls is slower, it could be a sign that it is simply using too much concurrency and getting bogged down in contention. |
findleyr
added
the
WaitingForInfo
|
I'm sorry that my expression has caused misunderstanding.
|
Ok, well that should help. Did you notice any improvement in gopls behavior? |
|
All that I have said above is the first time opening after clearing the gopls cache. The memory will not decrease so much unless I exit and reopen VSCode, and when opened for the second time, the memory will be much smaller than when opened for the first time. In the scene after the code has been cropped, it will first used ~5g memory, during use, it will gradually increase. |
Yes, gopls reduced the memory used for the first index project from 45G to 10G. |
5.16.mp4I think the indexing time is quite long. my computer is apple m3 pro. and after indexing, the memory usage is decrease to 34g from 47g
|
seankhliao
removed
the
WaitingForInfo
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
gopls version
go env
What did you do?
Open project in vscode, when first time open, it will consume nearly 40gb memory.
it can be reproduced if clean gopls cache and open the program again:
VSCode says it
Analyzing Dependencies.What did you see happen?
and the UI of VSCode is very laggy
What did you expect to see?
Normal memory consumption.
Editor and settings
Logs
and cannot see the
gopls.1234-5GiB-withnames.zipfile in $TMPDIRAs it is the company's code, I handled some sensitive information in the log.
gopls.log
The text was updated successfully, but these errors were encountered: