Distributor memory usage is high

[jakirpatel]

Describe the bug
We have on-premise cortex cluster. Currently we have 15 distributors and 15 ingesters. Currently 1 distributor is behaving weird with memory consumption reaching 100% of its limit memory.

Version
Cortex 1.11.0

Expected behavior
Distributor should run without OOM.

Environment:

• Infrastructure: Kubernetes on bare metal managed through Rancher.
• Deployment tool: Custom created with kustomize.

Storage Engine

• Blocks - Google Block Storage.

Additional Context

1. GOGC : We have setup GOGC values for both distributor and the ingester. The ingester does have the GOGC=30 and distributor does have the GOGC=80.

I think somehow the when the ingester GC happens we are seeing this impact. Is there any way to improve the situation.

[alvinlin123]

Hi, is your Prometheus sending HA labels? If so, this PR fixes a memory leak: #4739

If this is not the case, would you mind taking a heap profile?

[TaylorMutch]

FWIW updating to 1.13.0-rc.0 resolved a lot of my memory problems, as @alvinlin123 suggested.

[alvinlin123]

Good to know @TaylorMutch :-)