Skip to content
This repository was archived by the owner on Sep 30, 2024. It is now read-only.

It is not possible to alert on gitserver memory usage, possibly affects other Go services #12685

Open
emidoots opened this issue Aug 3, 2020 · 1 comment
Assignees
Labels
team/delivery Delivery team

Comments

@emidoots
Copy link
Member

emidoots commented Aug 3, 2020

Because of the way the Go memory allocator and gitserver work together, it uses up nearly all memory assigned to it regardless of the amount. This makes it almost impossible to alert on resulting in us removing the alerts for it.

The problem here is effectively that:

  1. RSS and VSS are very different topics, gitserver/Go use very little RSS but keep VSS around forever, basically.

image

  1. Go only cares about RSS, and treats VSS as effectively free - but in reality that is NOT true in k8s because cgroups OOM kills based on VSS.

  2. Go is not aware of VSS limits, like the ones we impose via k8s/Docker memory limits and does not appear intent on supporting that any time soon.

TODO: @slimsag resolve the issue in our Go services and/or write a blog post detailing/explaining the situation which has led to a lot of confusion elsewhere, including outside Sourcegraph.

@github-actions
Copy link
Contributor

Heads up @davejrt @ggilmore @dan-mckean @caugustus-sourcegraph @StephanX - the "team/delivery" label was applied to this issue.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
team/delivery Delivery team
Projects
None yet
Development

No branches or pull requests

3 participants