Lower or control memory usage #337

jirfag · 2018-12-26T14:16:31Z

Users complain about too much memory usage exceeding 8gb

lopezator · 2019-01-03T13:45:30Z

Related to #342 and pointers usage on latest commit.

We could dig more, sure, but it's a start.

jirfag · 2019-03-31T19:40:52Z

Hi!
A current solution may be to configure GOGC: it can trade memory to time and decrease memory usage in ~2 times. I've shown some results here.

jirfag · 2019-03-31T19:42:37Z

mem.pdf
Also, I've profiled memory of golangci-lint: PDF with results is attached. We can see that type checking consumes the most of memory. I have some ideas on how to fix it but they are all too costly to implement.

alexec · 2019-07-10T16:35:27Z

We have regular intermittent build failures due to high memory usage. It would be good to reduce usage.

Currently: we're using...

GOGC=20 golangci-lint run --fix --verbose --concurrency 1 --deadline 3m0s

See https://github.com/golangci/golangci-lint#memory-usage-of-golangci-lint

The crossplane jenkins worker node only has 10gb of RAM. With 1.17.1 of golangci-lint we regularly see 10gb consumed while linting in local environments. The changes suggested here appear to improve the overhead. golangci/golangci-lint#337 (comment) Prior to the bump to 1.17.1, golangci-lint was taking 6-8GB of RAM and completing in 10s. With the reduction in overhead to golangci-lint that this PR introduces, linting consumes between 8-9GB of ram and completes in 30s using the same environment. The intent here is to continue using latest golangci-lint with the current Jenkins configuration, sacrificing some time. Additional measure can be made, such as using `--fast` tests only or further increasing the LINT_GOGC value. https://github.com/golangci/golangci-lint#configuration https://github.com/golangci/golangci-lint#memory-usage-of-golangci-lint Signed-off-by: Marques Johansson <[email protected]>

howardjohn · 2019-09-17T15:20:46Z

istio/istio repo uses 32gb to lint: INFO Memory: 218 samples, avg is 18237.6MB, max is 31795.7MB

Set analysis pass results to nil early to garbage collect them soon. Memory can be reduced for the following linters: - staticcheck - stylecheck - gosimple - govet - bodyclose - any future go/analysis linter Relates: #712, #634, #628, #598, #509, #483, #337

daxmc99 · 2019-09-23T19:11:39Z

👍
Looking forward to testing out master

jirfag · 2019-09-23T19:28:47Z

I've merged a fix reducing memory usage of go/analysis linters (staticcheck, gosimple, stylecheck, bodyclose, govet) 5x. Please, check it in a master branch.

gracenoah · 2019-09-24T11:53:17Z

😍

bodyclose:   14.4 GB / 1m 20s -> 1.2 GB / 41s
govet:       15.5 GB / 1m 25s -> 1.8 GB / 30s
gosimple      5.9 GB / 49 s   -> 1.5 GB / 40s
staticcheck:  7.2 GB / 1m 1s  -> 1.7 GB / 1m 52s
govet:       15.5 GB / 1m 25s -> 1.8 GB / 30s
stylecheck: didn't run because statickcheck failed, but I'd expect similar numbers so I didn't test it

staticcheck seems to have slowed down, but it caught new errors, so maybe it was upgraded on master.

gracenoah · 2019-09-24T11:59:54Z

It's still using 29 GB of memory when running with all linters enabled rather than enabling 1 linter at a time. Is that expected? Is it just a side effect of Go's GC and it wouldn't use that much memory if it wasn't available?

jirfag · 2019-09-24T13:11:58Z

@gracenoah thank you for comparison,
I'am continuously reducing memory consumption: it's my top priority now. Is the repo rhere are you testing open-source?

gracenoah · 2019-09-24T14:48:26Z

It's not, sorry. It's an internal monorepo. If you want me to test some proposed performance changes, you can email me at [email protected] but that's all I can do.

jirfag · 2019-09-25T06:55:55Z

I've investigated more high memory consumption.

We can dramatically reduce memory consumption of all linters except unused, interfacer, unparam. To do it we need to run them by our go/analysis runner. Most of them require loader.Program or AST: we can mimic this interface from go/analysis's *packages.Package. It can take 1-3 weeks and reduce memory usage in the same way staticchecks memory dropped in v1.19.0.
The biggest question is what to do with unused, interfacer, unparam. I will research it later.

bflad · 2019-09-25T11:15:33Z

Hi @jirfag 👋 We continually have memory issues with https://github.com/terraform-providers/terraform-provider-aws if you're looking for an open source project to test. v1.19.1 still throws TravisCI out of memory errors for us with GOGC=30, which was working okay in v1.17.1. Our issue seems to have began in v1.18.0, similar to what was noted here: #731 (comment)

bmhatfield · 2019-09-25T13:29:09Z

Hi @jirfag - continuing the conversation here from #731!

A few hours ago, you wrote:

and reduce memory usage in the same way staticcheck's memory dropped in v1.19.0

But that is not the reality we are facing; in fact, we are experiencing the opposite (a huge increase in memory consumption that starts exactly at the commit claimed to improve things), as I demonstrated in #731.

Unfortunately, our codebase is not open-source, but I am happy to run experiments for you and report results back, as 6a979fb and beyond block our ability to upgrade golangci-lint.

bmhatfield · 2019-09-25T13:32:24Z

One additional note for folks struggling with this. For us, a large portion of memory consumption was coming from concurrent use of go's linker. We were previously able to reduce this significantly by setting GOGC=50, by limiting concurrency (to 4), and by running go vet separately from golangci-lint, but as of 6a979fb, that strategy is no longer effective.

jirfag · 2019-09-25T15:18:11Z

@bmhatfield I guess it was caused by large update of staticcheck. I will compare it in the end with 1.17

Run all linters per package. It allows unloading package data when it's processed. It dramatically reduces memory (and CPU because of GC) usage. Relates: #337

Get rid of AST cache: load AST when needed. Optimize memory allocations for go/analysis actions. Relates: #337

jirfag · 2019-09-30T19:46:07Z

Dramatically reduce memory usage #758 was merged.
I've made a bit more optimizations in reduce 1.5x memory usage on large repos on repeated runs #764
Unused is the only linter that eats a ton of memory because it can't work incrementally. You can disable it. You can make an issue in staticcheck GitHub repository about it: I think its optimization is possible.
I'm going to close this issue after merging reduce 1.5x memory usage on large repos on repeated runs #764 Please, write your comments if you think we need to fix something with memory before that.

Get rid of AST cache: load AST when needed. Optimize memory allocations for go/analysis actions. Relates: #337

pierrre · 2019-10-01T12:17:49Z

Do you plan to release another tag ?

fho · 2019-10-01T13:30:43Z

@jirfag thanks a lot for the recent optimizations!
Golangci-lint (df4f676) now only uses 68MB RAM on my machine.
Before it used all my 16GB and freezed my machine. :-)

jirfag · 2019-10-01T15:22:41Z

@fho I'm glad to hear that :)
@pierrre yep, it will be included in v1.20.0 in 1 week I think.

The latest version of golangci-lint is running slower and runs out of memory. See golangci/golangci-lint#337 for ongoing work on improving it. Triple the deadline to allow more time, and only run a single thread for testing to avoid running out of memory. Also make the garbage collector more aggressive to avoid holding onto excess memory. Signed-off-by: Michael Smith <[email protected]>

ryboe · 2019-10-07T18:12:39Z

❤️you so much @jirfag

jirfag · 2019-10-08T18:49:03Z

https://github.com/golangci/golangci-lint/releases/tag/v1.20.0 includes all memory optimizations

jirfag · 2019-10-08T18:49:27Z

@y0ssar1an thank you :)

…, add exclude-rules for unexpected SA1019 issues, replace deprecated deadline with timeout Reference: #10236 Reference: golangci/golangci-lint#337 (comment)

See golangci-lint memory improvement described at golangci/golangci-lint#337 which may help with problem on travis CI server when testing previous commit i.e. golangci-lint run --timeout 30m fatal error: runtime: out of memory

bodqhrohro · 2020-12-04T19:50:39Z

Even with GOGC=1, golangci-lint still uses more than 3 GB of RAM when analyzing Dendrite :C

alexec mentioned this issue Jul 10, 2019

memory consumption eruption for large golang file #530

Closed

displague mentioned this issue Aug 1, 2019

reduce golangci-lint overhead for jenkins worker upbound/build#68

Closed

jirfag mentioned this issue Sep 23, 2019

Drop memory usage of go/analysis linters 5x #725

Merged

jirfag mentioned this issue Sep 23, 2019

2x more memory than gometalinter #509

Closed

jirfag mentioned this issue Sep 23, 2019

Internal error, out of memory #483

Closed

jirfag mentioned this issue Sep 25, 2019

Significant/breaking increase in memory usage in 1.19 #731

Closed

jirfag added the topic: memory Huge memory consumption label Sep 25, 2019

This was referenced Sep 25, 2019

golangci-lint run Killed go1.13 #710

Closed

Out Of Memory failures for staticcheck on travis (GOGC=5, concurrency=1) #712

Closed

jirfag added this to the v1.20.0 milestone Sep 25, 2019

tpounds mentioned this issue Sep 25, 2019

Too much memory consumption, especially linter using ssa package. Memory: 811 samples, avg is 10074.1MB, max is 30712.9MB #634

Closed

jirfag added a commit that referenced this issue Sep 30, 2019

dramatically reduce memory usage (#758)

95ec0cf

Run all linters per package. It allows unloading package data when it's processed. It dramatically reduces memory (and CPU because of GC) usage. Relates: #337

jirfag added a commit that referenced this issue Sep 30, 2019

reduce 1.5x memory usage on large repos on repeated runs

faa8229

Get rid of AST cache: load AST when needed. Optimize memory allocations for go/analysis actions. Relates: #337

jirfag mentioned this issue Sep 30, 2019

reduce 1.5x memory usage on large repos on repeated runs #764

Merged

jirfag added a commit that referenced this issue Oct 1, 2019

reduce 1.5x memory usage on large repos on repeated runs (#764)

df4f676

Get rid of AST cache: load AST when needed. Optimize memory allocations for go/analysis actions. Relates: #337

jirfag closed this as completed Oct 1, 2019

siggy mentioned this issue Oct 1, 2019

Fix golangci-lint OOM linkerd/linkerd2#3288

Open

tpounds mentioned this issue Oct 1, 2019

Make incremental program loading #92

Closed

tpounds mentioned this issue Oct 4, 2019

Bazel build hangs on with 100% CPU #312

Closed

alexec mentioned this issue Oct 7, 2019

Upgrade golangci-lint to lower memory version 1.20.0 argoproj/argo-cd#2433

Closed

bflad mentioned this issue Oct 8, 2019

Update module golangci/golangci-lint to v1.20.0 hashicorp/terraform-provider-aws#10420

Merged

1 task

corneliusweig mentioned this issue Oct 8, 2019

Bump golangci-lint to v1.20.0 kubernetes/minikube#5572

Merged

supereagle mentioned this issue Oct 24, 2019

Update golangci-lint version for memory usage caicloud/cyclone#1315

Closed

k1eran mentioned this issue Jan 21, 2020

Add Junos 19.3R1 YANG and generated .go files onosproject/onos-config#1048

Closed

vboulineau mentioned this issue Jan 24, 2020

Increase CircleCI VM Size for unit tests DataDog/datadog-agent#4784

Merged

ritazh mentioned this issue Mar 6, 2020

Fix golangci-lint memory usage open-policy-agent/gatekeeper#511

Merged

ldez mentioned this issue Mar 23, 2020

Increased memory usage in 1.24.0 causing out-of-memory #994

Closed

3 tasks

bobcatfish mentioned this issue Mar 23, 2020

Ongoing GOLANGCI-LINT timeouts failing PRs and breaking nightly release tektoncd/plumbing#241

Closed

ankitm123 mentioned this issue May 5, 2020

refactor: add missing error checks jenkins-x/jx#7079

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lower or control memory usage #337

Lower or control memory usage #337

jirfag commented Dec 26, 2018

lopezator commented Jan 3, 2019

jirfag commented Mar 31, 2019

jirfag commented Mar 31, 2019

alexec commented Jul 10, 2019 •

edited

Loading

howardjohn commented Sep 17, 2019

daxmc99 commented Sep 23, 2019

jirfag commented Sep 23, 2019

gracenoah commented Sep 24, 2019

gracenoah commented Sep 24, 2019

jirfag commented Sep 24, 2019

gracenoah commented Sep 24, 2019

jirfag commented Sep 25, 2019

bflad commented Sep 25, 2019 •

edited

Loading

bmhatfield commented Sep 25, 2019

bmhatfield commented Sep 25, 2019 •

edited

Loading

jirfag commented Sep 25, 2019

jirfag commented Sep 30, 2019

pierrre commented Oct 1, 2019

fho commented Oct 1, 2019

jirfag commented Oct 1, 2019

ryboe commented Oct 7, 2019

jirfag commented Oct 8, 2019

jirfag commented Oct 8, 2019

bodqhrohro commented Dec 4, 2020

Lower or control memory usage #337

Lower or control memory usage #337

Comments

jirfag commented Dec 26, 2018

lopezator commented Jan 3, 2019

jirfag commented Mar 31, 2019

jirfag commented Mar 31, 2019

alexec commented Jul 10, 2019 • edited Loading

howardjohn commented Sep 17, 2019

daxmc99 commented Sep 23, 2019

jirfag commented Sep 23, 2019

gracenoah commented Sep 24, 2019

gracenoah commented Sep 24, 2019

jirfag commented Sep 24, 2019

gracenoah commented Sep 24, 2019

jirfag commented Sep 25, 2019

bflad commented Sep 25, 2019 • edited Loading

bmhatfield commented Sep 25, 2019

bmhatfield commented Sep 25, 2019 • edited Loading

jirfag commented Sep 25, 2019

jirfag commented Sep 30, 2019

pierrre commented Oct 1, 2019

fho commented Oct 1, 2019

jirfag commented Oct 1, 2019

ryboe commented Oct 7, 2019

jirfag commented Oct 8, 2019

jirfag commented Oct 8, 2019

bodqhrohro commented Dec 4, 2020

alexec commented Jul 10, 2019 •

edited

Loading

bflad commented Sep 25, 2019 •

edited

Loading

bmhatfield commented Sep 25, 2019 •

edited

Loading