runtime: `MADV_COLLAPSE` causes production performance issues on Linux

[mknyszek]

A Google production service experienced a performance regression with the Go runtime's use of MADV_COLLAPSE. We've narrowed down the issue to exactly that call. We suspect the issue is that MADV_COLLAPSE can go into direct reclaim while we're holding the heap lock. We suspect this issue is more widely applicable.

For now, let's roll back uses of MADV_COLLAPSE. We can revisit this in the future, but our current policy is almost certainly too aggressive given the costs.

[mknyszek]

@gopherbot Please open a backport issue for Go 1.21.

This issue has no workaround and can sometimes cause a significant performance regression with no workaround. The fix is small and safe (this codepath does not affect correctness).

[gopherbot]

Backport issue(s) opened: #63335 (for 1.21).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

[gopherbot]

Change https://go.dev/cl/531816 mentions this issue: runtime: don't eagerly collapse hugepages

[gopherbot]

Change https://go.dev/cl/532117 mentions this issue: runtime: delete hugepage tracking dead code

[mknyszek]

I accidentally auto-submitted CL 531816 with a fairly vague commit message. Here's the full intended commit message.

    MADV_COLLAPSE can go into direct reclaim, but we call it with the heap
    lock held. This means that the process could end up stalled fairly
    quickly if just one allocating goroutine ends up in the madvise call, at
    least until the madvise(MADV_COLLAPSE) call returns. A similar issue
    occurred with madvise(MADV_HUGEPAGE), because that could go into direct
    reclaim on any page fault for MADV_HUGEPAGE-marked memory.

    My understanding was that the calls to madvise(MADV_COLLAPSE) were
    fairly rare, and it's "best-effort" nature prevented it from going into
    direct reclaim often, but this was wrong. It tends to be fairly
    heavyweight even when it doesn't end up in direct reclaim, and it's
    almost certainly not worth it.

    Disable it until further notice and let the kernel fully dictate
    hugepage policy. The updated scavenger policy is still more hugepage
    friendly by delaying scavening until hugepages are no longer densely
    packed, so we don't lose all that much.

    The Sweet benchmarks show a minimal difference. A couple less realistic
    benchmarks seem to slow down a bit; they might just be getting unlucky
    with what the kernel decides to back with a huge page. Some benchmarks
    on the other hand improve. Overall, it's a wash.

    name                  old time/op            new time/op            delta
    BiogoIgor                        13.1s ± 1%             13.2s ± 2%    ~     (p=0.182 n=9+10)
    BiogoKrishna                     12.0s ± 1%             12.1s ± 1%  +1.23%  (p=0.002 n=9+10)
    BleveIndexBatch100               4.51s ± 4%             4.56s ± 3%    ~     (p=0.393 n=10+10)
    EtcdPut                         20.2ms ± 4%            19.8ms ± 2%    ~     (p=0.079 n=10+9)
    EtcdSTM                          109ms ± 3%             111ms ± 3%  +1.63%  (p=0.035 n=10+10)
    GoBuildKubelet                   31.2s ± 1%             31.3s ± 1%    ~     (p=0.780 n=9+10)
    GoBuildKubeletLink               7.77s ± 0%             7.81s ± 2%    ~     (p=0.237 n=8+10)
    GoBuildIstioctl                  31.8s ± 1%             31.7s ± 0%    ~     (p=0.136 n=9+9)
    GoBuildIstioctlLink              7.88s ± 1%             7.89s ± 1%    ~     (p=0.720 n=9+10)
    GoBuildFrontend                  11.7s ± 1%             11.8s ± 1%    ~     (p=0.278 n=10+9)
    GoBuildFrontendLink              1.15s ± 4%             1.15s ± 5%    ~     (p=0.387 n=9+9)
    GopherLuaKNucleotide             19.7s ± 1%             20.6s ± 0%  +4.48%  (p=0.000 n=10+10)
    MarkdownRenderXHTML              194ms ± 3%             196ms ± 3%    ~     (p=0.356 n=9+10)
    Tile38QueryLoad                  633µs ± 2%             629µs ± 2%    ~     (p=0.075 n=10+10)

    name                  old average-RSS-bytes  new average-RSS-bytes  delta
    BiogoIgor                       69.2MB ± 3%            68.4MB ± 1%    ~     (p=0.190 n=10+10)
    BiogoKrishna                    4.40GB ± 0%            4.40GB ± 0%    ~     (p=0.605 n=9+9)
    BleveIndexBatch100               195MB ± 3%             195MB ± 2%    ~     (p=0.853 n=10+10)
    EtcdPut                          107MB ± 4%             108MB ± 3%    ~     (p=0.190 n=10+10)
    EtcdSTM                         91.6MB ± 5%            92.6MB ± 4%    ~     (p=0.481 n=10+10)
    GoBuildKubelet                  2.26GB ± 1%            2.28GB ± 1%  +1.22%  (p=0.000 n=10+10)
    GoBuildIstioctl                 1.53GB ± 0%            1.53GB ± 0%  +0.21%  (p=0.017 n=9+10)
    GoBuildFrontend                  556MB ± 1%             554MB ± 2%    ~     (p=0.497 n=9+10)
    GopherLuaKNucleotide            39.0MB ± 3%            39.0MB ± 1%    ~     (p=1.000 n=10+8)
    MarkdownRenderXHTML             21.2MB ± 2%            21.4MB ± 3%    ~     (p=0.190 n=10+10)
    Tile38QueryLoad                 5.99GB ± 2%            6.02GB ± 0%    ~     (p=0.243 n=10+9)

    name                  old peak-RSS-bytes     new peak-RSS-bytes     delta
    BiogoIgor                       90.2MB ± 4%            89.2MB ± 2%    ~     (p=0.143 n=10+10)
    BiogoKrishna                    4.49GB ± 0%            4.49GB ± 0%    ~     (p=0.190 n=10+10)
    BleveIndexBatch100               283MB ± 8%             274MB ± 6%    ~     (p=0.075 n=10+10)
    EtcdPut                          147MB ± 4%             149MB ± 2%  +1.55%  (p=0.034 n=10+8)
    EtcdSTM                          117MB ± 5%             117MB ± 4%    ~     (p=0.905 n=9+10)
    GopherLuaKNucleotide            44.9MB ± 1%            44.6MB ± 1%    ~     (p=0.083 n=8+8)
    MarkdownRenderXHTML             22.0MB ± 8%            22.1MB ± 9%    ~     (p=0.436 n=10+10)
    Tile38QueryLoad                 6.24GB ± 2%            6.29GB ± 2%    ~     (p=0.218 n=10+10)

    name                  old peak-VM-bytes      new peak-VM-bytes      delta
    BiogoIgor                       1.33GB ± 0%            1.33GB ± 0%    ~     (p=0.504 n=10+9)
    BiogoKrishna                    5.77GB ± 0%            5.77GB ± 0%    ~     (p=1.000 n=10+9)
    BleveIndexBatch100              3.53GB ± 0%            3.53GB ± 0%    ~     (p=0.642 n=10+10)
    EtcdPut                         12.1GB ± 0%            12.1GB ± 0%    ~     (p=0.564 n=10+10)
    EtcdSTM                         12.1GB ± 0%            12.1GB ± 0%    ~     (p=0.633 n=10+10)
    GopherLuaKNucleotide            1.26GB ± 0%            1.26GB ± 0%    ~     (p=0.297 n=9+10)
    MarkdownRenderXHTML             1.26GB ± 0%            1.26GB ± 0%    ~     (p=0.069 n=10+10)
    Tile38QueryLoad                 7.47GB ± 2%            7.53GB ± 2%    ~     (p=0.280 n=10+10)

    name                  old p50-latency-ns     new p50-latency-ns     delta
    EtcdPut                          19.8M ± 5%             19.3M ± 3%  -2.74%  (p=0.043 n=10+9)
    EtcdSTM                          81.4M ± 4%             83.4M ± 4%  +2.46%  (p=0.029 n=10+10)
    Tile38QueryLoad                   241k ± 1%              240k ± 1%    ~     (p=0.393 n=10+10)

    name                  old p90-latency-ns     new p90-latency-ns     delta
    EtcdPut                          30.4M ± 5%             30.6M ± 5%    ~     (p=0.971 n=10+10)
    EtcdSTM                           222M ± 3%              226M ± 4%    ~     (p=0.063 n=10+10)
    Tile38QueryLoad                   687k ± 2%              691k ± 1%    ~     (p=0.173 n=10+8)

    name                  old p99-latency-ns     new p99-latency-ns     delta
    EtcdPut                          42.3M ±10%             41.4M ± 7%    ~     (p=0.353 n=10+10)
    EtcdSTM                           486M ± 7%              487M ± 4%    ~     (p=0.579 n=10+10)
    Tile38QueryLoad                  6.43M ± 2%             6.37M ± 3%    ~     (p=0.280 n=10+10)

    name                  old ops/s              new ops/s              delta
    EtcdPut                          48.6k ± 3%             49.5k ± 2%    ~     (p=0.065 n=10+9)
    EtcdSTM                          9.09k ± 2%             8.95k ± 3%  -1.56%  (p=0.045 n=10+10)
    Tile38QueryLoad                  28.4k ± 1%             28.6k ± 1%  +0.87%  (p=0.016 n=9+10)

[gopherbot]

Change https://go.dev/cl/532255 mentions this issue: [release-branch.go1.21] runtime: don't eagerly collapse hugepages