Skip to content
This repository was archived by the owner on Sep 30, 2024. It is now read-only.
This repository was archived by the owner on Sep 30, 2024. It is now read-only.

It is not possible to alert on gitserver memory usage, possibly affects other Go services #12685

Open
@emidoots

Description

@emidoots

Because of the way the Go memory allocator and gitserver work together, it uses up nearly all memory assigned to it regardless of the amount. This makes it almost impossible to alert on resulting in us removing the alerts for it.

The problem here is effectively that:

  1. RSS and VSS are very different topics, gitserver/Go use very little RSS but keep VSS around forever, basically.

image

  1. Go only cares about RSS, and treats VSS as effectively free - but in reality that is NOT true in k8s because cgroups OOM kills based on VSS.

  2. Go is not aware of VSS limits, like the ones we impose via k8s/Docker memory limits and does not appear intent on supporting that any time soon.

TODO: @slimsag resolve the issue in our Go services and/or write a blog post detailing/explaining the situation which has led to a lot of confusion elsewhere, including outside Sourcegraph.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions