runtime: increase in memory usage and different GC behavior upgrading from 1.13.10 to 1.14.2

[BorisKozo]

Note that I read carefully through #37927 and #35112 but it seems those issues were handled already.

What version of Go are you using (go version)?

Upgrade 1.13.10 -> 1.14.2

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

Windows 10 (reproduces on other Windows versions as well)

What did you do?

Run my application memory benchmark

What did you expect to see?

Same or better results as in 1.13.10

What did you see instead?

Worse results

I have to add that our application is resource critical so every increase in memory or CPU usage raises eyebrows immediately.

I compiled our application with both GO 1.13.10 and 1.14.2 and the CPU and memory behavior are inconsistent between the two versions as the later consumes more memory (enough to trigger the benchmark failure). The behavior of the memory release is also different (see charts below) as there is no visible de-allocation of memory in 1.14.2.
I also tried changing the GOGC to various values but even at 10 the memory did not reach the behavior of 1.13.10 and CPU usage was considerable

Initial run compiled with GO 1.13.10

Same code was compiled and run on the same machine with GOGC values as follows:

GOGC = 10

GOGC = 50

GOGC = 100

GOGC = 200

GOGC = 500

in Issue #37525 @mknyszek mentions that another issue will be opened but I could not find the new issue.

Thanks.

[agnivade]

Would you be able to share the output of setting GODEBUG=gctrace=1,gcpacertrace=1 in 1.13.10 vs 1.14.2 ? That would help us a lot in the investigation. Thanks.

@aclements @mknyszek

[BorisKozo]

Hi, I generated the traces:
1.13.10
trace_1_13_10.txt

and
1.14.2
trace_1_14_2.txt

Look at the chart for 1.14.2, doesn't appear to be releasing anything and I am running with GOGC=100

[mknyszek]

Regarding memory release, #38617 brought our attention to a bug on Windows where in Go 1.14 the background scavenger wasn't running at all (the synchronous/eager scavenger and debug.FreeOSMemory should still work though). It's not clear to me yet that this is exactly the same problem, but try patching Go with https://golang.org/cl/229997 and see if it helps. Based on the traces you provided, I could see it helping.

Even with that patch, you may not be seeing the same log as in Go 1.13.10 with GODEBUG=gctrace=1 because we moved the scavenge trace under its own flag, which prints once per GC cycle. The flag for Go 1.14+ is GODEBUG=scavtrace=1.

[networkimprov]

@gopherbot add OS-Windows

[BorisKozo]

@mknyszek I compiled master at a1ffbe9
and it seems the problem has been resolved.

Thank you very much!
Which release is this fix planned to?

[networkimprov]

I posted a backport request. If there's no intervening security update, it will land in 1.14.3.

EDIT: #38856

[BorisKozo]

Thank you

…

On Mon, 4 May 2020 at 22:01 Liam ***@***.***> wrote: I posted a backport request. If there's no intervening security update, it will land in 1.14.3. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#38827 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOTAGWDMDRNDJXKT5A67KTRP4GG5ANCNFSM4MYBZD6A> .