runtime: MADV_HUGEPAGE causes stalls when allocating memory

[dominikh]

Environment: linux/amd64

I've bisected stalls in one of my applications to 8fa9e3b — after discussion with @mknyszek, the stalls seem to be caused by Linux directly reclaiming pages, and taking significant time to do so (100+ ms in my case.)

The direct reclaiming is caused by the combination of Go setting memory as MADV_HUGEPAGE and Transparent Huge Pages being configured as such on my system (which AFAICT is a NixOS default; I don't recall changing this:)

$ cat /sys/kernel/mm/transparent_hugepage/enabled 
always [madvise] never
$ cat /sys/kernel/mm/transparent_hugepage/defrag 
always defer defer+madvise [madvise] never

In particular, the madvise setting for defrag has the following effect:

will enter direct reclaim like always but only for regions that are have used madvise(MADV_HUGEPAGE). This is the default behaviour.

with always meaning

means that an application requesting THP will stall on allocation failure and directly reclaim pages and compact memory in an effort to allocate a THP immediately. This may be desirable for virtual machines that benefit heavily from THP use and are willing to delay the VM start to utilise them

It seems to me that one of the reasons for setting MADV_HUGEPAGE is to undo setting MADV_NOHUGEPAGE and that there is no other way to do that.

[mknyszek]

Yeah, this is unfortunate. The immediate workaround is to set /sys/kernel/mm/transparent_hugepage/defrag to defer.

Here's one idea to resolve this: only use MADV_HUGEPAGE and MADV_NOHUGEPAGE if /sys/kernel/mm/transparent_hugepage/defrag is defer. Except the problem with that is khugepaged can absolutely back memory not marked MADV_NOHUGEPAGE as long as /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none is non-zero. Part of the reason for the explicit MADV_NOHUGEPAGE is to prevent this background coalescing, because max_ptes_none is 511 by default.

One other idea is to use the new MADV_COLLAPSE in place of MADV_HUGEPAGE, and skip MADV_NOHUGEPAGE entirely (MADV_DONTNEED already breaks up huge pages). I'm pretty sure this means we'd only enter reclaim on the thread invoking the collapse. (And presumably that won't block threads from accessing memory until the backing pages are actually swapped out, akin to how I assume khugepaged works.) However with this strategy we're still opening ourselves up to that again unless everyone sets max_ptes_none to zero (like TCMalloc recommends).

Linux defaults are working against us here. They're really not great!

The best, quickest fix would be a way to clear the VM_NOHUGEPAGE flag without also setting MADV_HUGEPAGE, but I don't see a way to do that at the moment. I'll keep digging.

[mknyszek]

@randall77 Pointed out to me that we might be able to mitigate the issue by eagerly accessing the memory region we just set to MADV_HUGEPAGE. This happens either when growing the heap (very rarely) or on a background goroutine, where it's probably more OK to stall. If we can move the stall off of the critical path, that might be enough.

[mknyszek]

As an update here, we've been discussing this on the Gophers Slack. @dominikh sent me a smaller reproducer (https://go.dev/play/p/4d11Jc5nNDi.go) that produces some fairly large outliers on his system, but I've been unable to reproduce in two different VMs so far, after setting the hugepage sysfs parameters to align.

@dominikh noted that if he shuts down a whole bunch of applications then it seemingly goes away. He reported that he can reproduce while stress testing with the tool https://github.com/resurrecting-open-source-projects/stress. I've been trying with that, but also with a homebrew GC stress test. We've also been making sure /proc/buddyinfo indicates that most of the unfragmented memory is drained. For me, I've only been able to get that to happen with the homebrew GC stress test.

In the end, I can produce high latencies, but the distribution is approximately the same between Go 1.20 and tip-of-tree.

I'm not yet sure what the difference is. I've been trying on Linux 5.15 and 6.3, @dominikh is using 6.2.

[dominikh]

Meanwhile I can reliably reproduce the issue with Go at master, and not at all with Go 1.20. In fact, seeing such high latencies with Go 1.20 seems particularly weird, as it's not actually making use of huge pages much at all.

The problem is very sensitive to the amount of memory fragmentation, which can make it difficult to trigger. There have to be few enough (or none) allocations available for huge pages, and "direct reclaim" must not be able to quickly merge pages, either.

I did end up having more luck with Michael's stress test than with stress. At this point, running 10 instances of Michael's stress test — compiled with Go master — followed by my reproducer, leads to a maximum pause of 1.29s when the reproducer is built with Go master, and ~20ms when built with Go 1.20.6, with the 20ms pauses likely being due to Linux scheduler pressure, as the stress test keeps all cores busy.

It's also worth noting that the stress test acts as an easier way of reproducing the problem, not a requirement. I originally reproduced the problem just by having a lot of typical, heavy desktop software open (Firefox with a significant amount of tabs, Discord, Slack), which left memory quite fragmented. The stress test is a much easier way of using up large allocations.

[mknyszek]

as it's not actually making use of huge pages much at all.

There are actually a few times Go 1.20 might call MADV_HUGEPAGE, but they're hard to predict. I suppose it's possible that those situations happen to come up in my attempts to reproduce? I could add logging.

Though, FWIW, I'm willing to believe that I'm just doing something wrong here in my attempt to reproduce this. The upshot is we have a way to reproduce it somewhere.

The trouble is there still doesn't seem like there's a clear path forward. I neglected to mention that the earlier suggestion of trying to eagerly force the "direct reclaim" didn't really change anything, unfortunately.

[mknyszek]

And I forgot to say, thank you for your time and effort in looking into this @dominikh!

[dominikh]

I don't think there's a hands-off solution that will make everyone happy.

If Go makes explicit use of transparent huge pages, then it opens itself up to all of the common issues with THP, such as compaction stalls, khugepaged going crazy, being sensitive to other programs running on the system, and so on¹. It effectively requires some users to 1) be aware of THP and 2) tweak their system configuration, either tuning or disabling THP.

On the other hand, not making use of transparent huge pages wastes performance for some workloads, and there would currently be no other way for users to make use of THP that doesn't involve avoiding Go's allocator altogether.

I don't think the problem described in this issue is unique to Go; it also haunts other allocators that make use of THP, and there doesn't seem to be a way to control THP precisely enough — we'll always have to deal with the fact that different Linux distributions ship different defaults, some of which work worse for us than others.

Specifically, the following two approaches seem to be impossible using the current APIs offered by the kernel:

• Only caring about preventing huge pages. It's impossible to prevent huge pages (MADV_NOHUGEPAGE) if one wishes to undo it later. Only MADV_HUGEPAGE can clear the MADV_NOHUGEPAGE flag, and that's too strong a signal: it will explicitly cause huge pages to be used on many Linux configurations.
• Explicitly using THP (via MADV_HUGEPAGE) while wanting to maintain bounded latency for allocating. Only the system configuration can control if failing allocations stall and defrag.

Personally, I don't think that the Go runtime knows enough about the workload, the system, or the requirements to decide whether stalling on allocations is acceptable, worth it, or detrimental. On the other hand, Go's focus is on server software, and maybe it's okay to assume that server environments have enough memory for our process, or are configured appropriately with regard to THP. However, Go is used for all kinds of applications, and run in environments where the user cannot change THP settings globally, so there should probably be a way to disable the use of THP when it's known to work poorly for the application, e.g. via a GODEBUG variable. This would be somewhat similar to GOGC, but do we really want to introduce another knob that most people won't be aware of and won't know how to determine when to use?

¹: I've likely only encountered this issue because most of system memory was already in use by the ZFS ARC — which does get dropped when needed, but apparently not to allow for pages to be merged into huge pages. There are probably other unique combinations leading to issues and Go cannot predict all of them.

[mknyszek]

I agree with your assessment of the situation. One additional question I have from you is when in your reproducer (small or big) does the stall happen? Is it close to application start? Does it happen many times while the application is running?

My hypothesis is that you're fairly likely to see this at process start, and then no more (once the huge page is installed). The reason is that the runtime calls MADV_HUGEPAGE on all new heap memory.

Furthermore, I've been poking around kernel mailing list messages and I'm even more convinced that MADV_HUGEPAGE is just the wrong hammer.

However, I do think MADV_COLLAPSE might really be the right hammer. Here's my reasoning:

• With MADV_COLLAPSE, we don't have to mark new memory. We only use it when we need it, which will generally be rarely and always on a background thread.
• Direct reclaim and direct compaction are operations that apply to the whole system. But my hypothesis is that it doesn't necessarily cause full-process or full-system stalls (it matters when moving physical memory around, but I suspect it doesn't just acquire a lock on the entire system or something), just thread stalls to fulfill the allocation request.
• MADV_COLLAPSE ignores every hugepage setting except MADV_NOHUGEPAGE. This is a good thing because it basically fully puts control of huge pages into the hands of the allocator.

Regarding the issue of khugepaged just backing memory returned to the OS via madvise, I'm coming to the conclusion that max_ptes_none=511 is just not a very good default for memory allocators and we shouldn't try to deal with it. TCMalloc indirectly recommends setting it to zero. I'm inclined to just give up and do the same. (MADV_NOHUGEPAGE also doesn't compose well with MADV_COLLAPSE (the latter respects the former).)

Linux seems to take the stance that everyone should just tune the huge page settings to their application. Unfortunately I thin that forces our hand for the most part.

I still believe the foundation 8fa9e3b is based on is sound. Specifically that most of the time, the Go heap really should just be backed by huge pages. We've got a first-fit allocator, and many objects are small. They pack densely into pages reaching up to the heap goal, at which point there may be some fragmentation, which the scavenger picks at.

Taking all of that into account, here's what I think we should do:

• Use MADV_COLLAPSE instead of sysHugePage when available.
• Don't call sysNoHugePage at all when the scavenger is running.
• Add a section to the optimization guide at https://go.dev/doc/gc-guide about configuring huge pages.

I admit the third point seems awfully specific, but there are way too many people out there trying to figure out what is going on with memory on their Linux system when it comes to transparent huge pages. If I can help that just a little bit, I think it's worthwhile.

These are small changes. I'll prototype this, benchmark it, and see what the effect is. I'll also share the patch with you @dominikh if you're up for trying your reproducer again.

[dominikh]

Is it close to application start? Does it happen many times while the application is running?

The stalls happen many times. Presumably every time the runtime has to allocate more memory from the OS.

My hypothesis is that you're fairly likely to see this at process start, and then no more (once the huge page is installed). The reason is that the runtime calls MADV_HUGEPAGE on all new heap memory.

But few Go programs allocate all their memory at process start? Heaps grow over time. Or they shrink, return the memory to the OS, and grow again.

I'll also share the patch with you @dominikh if you're up for trying your reproducer again.

Happily.

[mknyszek]

But few Go programs allocate all their memory at process start? Heaps grow over time. Or they shrink, return the memory to the OS, and grow again.

It's true that they don't allocate all their memory at process start, I failed to say what I meant. What I meant was that eventually in some steady-state you'll stop seeing new stalls because the heap will have stretched to its peak size in terms of mapped memory. Because we don't unmap heap memory, shrinking and regrowing the heap shouldn't cause new stalls in the current implementation, except if there's a substantial amount of time between shrink/regrowth.

I still think this is not very good, but I mainly wanted to confirm that what you were seeing was the former case (stalls on up-front heap growth) vs. the latter case (stalls from the scavenger setting MADV_HUGEPAGE again).

[dominikh]

I was concretely seeing stalls in a graphical application that (unfortunately) allocates when rendering frames, and the occurrence of stalls lasted long enough for me to debug a single process for several minutes.

The application allocates a significant amount of memory upfront when loading traces, but then allocates much smaller amounts of memory as it renders frames. This causes the heap to grow slowly, with no GC cycles because of the high GC target, but frequent stalls.

Because we don't unmap heap memory, shrinking and regrowing the heap shouldn't cause new stalls in the current implementation, except if there's a substantial amount of time between shrink/regrowth

Can you define "substantial amount of time" for me? I was under the impression that we returned memory to the OS every ~3 minutes via the background scavenger, and actively during allocations. That would mean that stalls affect bursty workloads, too, if they happen apart far enough, though this wouldn't apply to my concrete reproducers.

I can see how a steady workload can reach a steady state where we no longer need to ask the OS for new pages, but I agree that that's "still not very good."

[mknyszek]

Can you define "substantial amount of time" for me?

The conditions for MADV_HUGEPAGE are:

• The scavenger must have returned some memory in a candidate 4 MiB chunk in the past. For this to happen, the chunk must have been <96% occupied for at least one full GC cycle.
• The 4 MiB chunk must now be at least 96% occupied.

If you're not GCing frequently, the maximum time the cycle can take is 2 minutes to let the scavenger see the free memory, plus whatever time it takes for the scavenger to find the available memory, plus the time until enough allocations happen in that chunk to set it as MADV_HUGEPAGE again. Under memory pressure the "eager scavenger" is allowed to ignore the "full GC cycle" requirement. However, the eager scavenger should be rarer in Go 1.21 thanks to some additional hedging in the memory limit heap goal calculation.

Thanks for the detail on the application though, I think that narrows down the issue to heap growth (which you already knew). It also makes sense that since the heap grows relatively slowly, you just end up seeing this over and over.

I started working on a patch, but it occurs to me that MADV_COLLAPSE would still happen on an application thread, because it would happen at allocation time. :( This is still probably an OK start and will still reduce the chance of these stalls significantly.

[mknyszek]

Here's the CL: https://go.dev/cl/516795

If it works, I won't be surprised, given everything I now know about your application. Still, it would be good to confirm.

[dominikh]

The CL seems to fix the stalls in the minimal reproducer.

Go master latency

N 900000  sum 6.44549e+10  mean 71616.5  gmean 62407.8  std dev 821383  variance 6.7467e+11
     min 7223
   1%ile 16610
   5%ile 58507
  25%ile 60611
  median 63075
  75%ile 64758
  95%ile 71911
  99%ile 114225
     max 3.33635e+08
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢐⡂⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡖ 0.000

⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡛⠃⢆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇

⠠⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠴⠀⠀⠈⠲⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠄⠧ 0.000

⠈⠉⠉⠋⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠙⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠁

0                               100000

Go 1.20 latency

N 900000  sum 5.95848e+10  mean 66205.4  gmean 64138.2  std dev 21012.2  variance 4.41511e+08
     min 7495
   1%ile 16864
   5%ile 60048
  25%ile 62193
  median 63795
  75%ile 66009
  95%ile 72543
  99%ile 230538
     max 627926
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢐⡂⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡖ 0.000

⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡌⠸⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇

⠠⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠂⠀⠱⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠄⠧ 0.000

⠈⠉⠉⠉⠋⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠋⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠙⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠁

0                     100000                   200000

Go MADV_COLLAPSE latency

N 900000  sum 5.95614e+10  mean 66179.3  gmean 64054.9  std dev 21151.5  variance 4.47384e+08
     min 7034
   1%ile 15761
   5%ile 59926
  25%ile 62120
  median 63753
  75%ile 65908
  95%ile 73061
  99%ile 231009
     max 628989
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠰⡆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡖ 0.000

⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡊⢨⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇

⠠⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠃⠀⠳⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠄⠧ 0.000

⠈⠉⠉⠉⠋⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠋⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠋⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠁

0                     100000                   200000

I've also collected some key statistics about timing and memory usage.

Go master statistics

	User time (seconds): 57.41
	System time (seconds): 6.09
	Percent of CPU this job got: 101%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:02.68
	Maximum resident set size (kbytes): 11953768
	Minor (reclaiming a frame) page faults: 704619
	Voluntary context switches: 14590
	Involuntary context switches: 300

Go 1.20 statistics

	User time (seconds): 58.06
	System time (seconds): 3.51
	Percent of CPU this job got: 101%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:00.95
	Maximum resident set size (kbytes): 11945216
	Minor (reclaiming a frame) page faults: 3000881
	Voluntary context switches: 10791
	Involuntary context switches: 196

Go collapse statistics

	User time (seconds): 58.31
	System time (seconds): 3.57
	Percent of CPU this job got: 100%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:01.28
	Maximum resident set size (kbytes): 11944192
	Minor (reclaiming a frame) page faults: 2989475
	Voluntary context switches: 14209
	Involuntary context switches: 219

However, looking at the number of huge pages used by the process, the
minimal reproducer no longer uses any huge pages, so I don't think it's
able to measure the impact of MADV_COLLAPSE on long-running processes.

I've also tested it with my actual application: The CL manages to
eliminate the stalls here, too. It doesn't use any huge pages for the
first ~four minutes of the process's lifetime (Go 1.20 allocates 1.37
GiB worth of huge pages and Go master allocates 3.08 GiB worth, right at
the beginning, when we allocate large amounts of contiguous memory for
loading data.)

After ~4 minutes, the process allocates exactly 2048 KiB worth of huge
pages, even though there are 5 GiB worth of 2 MiB and 4 MiB buddies
available, before considering compaction/defragmentation.

I've left the process running for several more minutes, after which huge
pages usage jumped from 2048 KiB to 6144 KiB. That is, we managed to go
from 1 huge page to 3 huge pages. I stopped running the test at this
point.

Usage of huge pages was measured via the AnonHugePages field in /proc/<pid>/smaps¹.

¹: For anyone trying to reproduce the stutter, make sure not to be reading from smaps periodically while doing so. Acessing smaps itself can introduce stutter.

[mknyszek]

After ~4 minutes, the process allocates exactly 2048 KiB worth of huge
pages, even though there are 5 GiB worth of 2 MiB and 4 MiB buddies
available, before considering compaction/defragmentation.

Given your THP settings, this is roughly what I would expect (huge pages are only allocated on MADV_HUGEPAGE). There are still a couple places left where we MADV_HUGEPAGE a single time when the heap exceeds 1 GiB that the patch doesn't address which is probably where that 6 MiB is coming from.

The 1.37 GiB from Go 1.20 is likely a result of a combination of the old MADV_HUGEPAGE/MADV_NOHUGEPAGE behavior and the scavenger creating holes (which are then filled) when the heap grows.

I'm benchmarking with a THP setting of always which seems to still allocate a decent number of huge pages.

One thing I'm curious about is whether the stall problem still exists if we MADV_COLLAPSE whenever the heap first gets dense. It's hard to tell from the documentation what will happen. On the one hand, MADV_COLLAPSE is described as "best-effort" while on the other hand it says "may enter direct reclaim" and I'm really not certain how strong the former is and if the latter is just generally leaving the door open on the implementation or if it's as aggressive as MADV_HUGEPAGE.

I'll send another patch, if you're still on-board for trying it out!

[mknyszek]

Here it is: https://go.dev/cl/516995, which is meant to be patched on top of https://go.dev/cl/516795.

[dominikh]

$ ~/prj/go-collapse/bin/go version              
go version devel go1.22-93e59ca7d2 Tue Aug 8 13:35:40 2023 +0000 linux/amd64

$ git log -2 --oneline 93e59ca7d2
93e59ca7d2 (HEAD) runtime: consider the heap as not backed by huge pages by default
3cc0d4c8c5 runtime: avoid MADV_HUGEPAGE for heap memory

Actual application: 2048 KiB of huge pages right away, 4096 KiB after several minutes. No stalls during normal use, but impossible for me to time my manual testing with the scavenger running.
Minimal reproducer: 2048 KiB of huge pages right away, 2048 KiB after several minutes. No stalls.

Output of strace on the minimal reproducer, for calls to madvise:

madvise(0x7fa0db931000, 33554432, MADV_NOHUGEPAGE) = 0
madvise(0x7fa0db71c000, 1048576, MADV_NOHUGEPAGE) = 0
[pid 2681659] madvise(0xc000000000, 4194304, MADV_COLLAPSE) = -1 EINVAL (Invalid argument)
[pid 2681661] madvise(0x7fa0dba00000, 31457280, MADV_HUGEPAGE) = 0

[mknyszek]

Output of strace on the minimal reproducer, for calls to madvise:

I was able to reproduce that and I figured out the issue. MADV_COLLAPSE requires at least one physical page to be faulted in for the region to work, and right now the runtime shuts it off permanently on any error. A reuploaded the CLs from earlier with a quick fix that just faults in a physical page and that makes MADV_COLLAPSE actually run.

[gopherbot]

Change https://go.dev/cl/516795 mentions this issue: runtime: avoid MADV_HUGEPAGE for heap memory

[mknyszek]

We've been discussing on the Gophers Slack. Summarizing (and skipping some of the wild goose chasing):

• https://go.dev/cl/516995 is not a good idea. Collapsing new memory for big allocations will almost always fail because it hasn't been faulted in yet. We'll just rely on the OS to back with hugepages at its leisure.
• The restriction around "a page must have been faulted previously" strongly suggests to me this feature is about repairing holes created by MADV_DONTNEED.
• Eagerly collapsing memory for big allocations is very likely to be undesirable. What if the user is relying on demand paging and is implementing some kind of sparse map? It's not exactly portable but this would kill that use-case. I updated my patch to not do that anymore by skipping the collapse if the allocation causing it is filling up an entire chunk.
• There's just no way to tell if madvise(MADV_COLLAPSE) failed because a hugepage region didn't contain a faulted-in page or if it's not supported. Oh well.

I think https://go.dev/cl/516795 might be the fix. I've asked @dominikh to run my homebrew GC stress test with GOMEMLIMIT=512MiB GOGC=off because I've previously used it to debug excessive scavenging calls with the memory limit, and I know that it still eagerly scavenges (and subsequently collapses pages) at least a good bit.

[mknyszek]

@gopherbot Please open a backport issue for Go 1.21.

This can cause unbounded stalls on Linux in some cases with no workaround.

[gopherbot]

Backport issue(s) opened: #62329 (for 1.21).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

[gopherbot]

Change https://go.dev/cl/523655 mentions this issue: [release-branch.go1.21] runtime: avoid MADV_HUGEPAGE for heap memory

[kevinconaway]

👋
We noticed after updating to Go 1.21.0 that some of our apps were using more off heap memory than others. There was a wide gap in between the reported container memory (kubernetes) and the process RSS or go mstats heap_sys value.

We figured that this issue might be related and indeed updating to Go 1.21.1 solves the issue for us but we aren't sure how the problem described here could contribute to larger amounts of retained off heap memory.

First, does the problem described here sound like it could also cause the issue that we are/were seeing? Second, do you have any suggestions for telemetry that we could look at to confirm? None of the usual suspects for us were showing anything apart from the aforementioned gap between the container memory and go heap memory

The graph below illustrates the gap, with go 1.21.0 running and then the same service being deployed with 1.20.7

[mknyszek]

@kevinconaway It certainly could be. Go 1.21.1, which went out today, fixes this issue for Go 1.21. You can give that a try.

If you want another way to check if you're affected, check the output of:

$ cat /sys/kernel/mm/transparent_hugepage/enabled 

where you're running your code. If it's madvise, then this is likely the issue you're looking for.

[kevinconaway]

Go 1.21.1 does indeed solve the issue for us.

another way to check if you're affected, check the output of:

Below is the output:

$ cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never

Are there any metrics that we could have helped us track this down further, or similar issues in the future? All we were able to tell was that the memory was being retained somewhere "off heap" but we had little visibility into what it was.

[mknyszek]

Go 1.21.1 does indeed solve the issue for us.

Glad to hear!

Are there any metrics that we could have helped us track this down further, or similar issues in the future? All we were able to tell was that the memory was being retained somewhere "off heap" but we had little visibility into what it was.

Unfortunately Linux doesn't provide a very good way to observe how much memory is going to huge pages. You can occasionally dump /proc/<pid>/smaps and total up the AnonHugePages count. Other than that, I don't think there's a lot you can do. :(

Fortunately, I don't think you'll have to worry about this being a problem from Go in the future. Now that we have a better understanding of the landscape of hugepage-related madvise syscalls, I don't think we'll be trying to explicitly back the heap with hugepages outside of specific cases and only a best-effort basis (like Go 1.21.1 now does).

[bboreham]

/proc/<pid>/smaps_rollup can do the totalling for you.

[gopherbot]

Change https://go.dev/cl/526615 mentions this issue: _content/doc: discuss transparent huge pages in the GC guide

[gopherbot]

Change https://go.dev/cl/531816 mentions this issue: runtime: don't eagerly collapse hugepages

[gopherbot]

Change https://go.dev/cl/532117 mentions this issue: runtime: delete hugepage tracking dead code

[gopherbot]

Change https://go.dev/cl/532255 mentions this issue: [release-branch.go1.21] runtime: don't eagerly collapse hugepages