Gitea depletes Ram + CPU

[JoeyWrk]

• Gitea version: Gitea version 1.12.2

• Git version: Git version 2.28-rc1

• Operating system: Ubuntu 20.04.1 LTS (on Virtual Machine (HyperV)

• Database :

  • [x ] MySQL (MariaDB 10.3.22)

• Can you reproduce the bug at https://try.gitea.io:

  • [ x] Not relevant

Description

Gitea uses up all system resources, even though all System Requirements are met.

I have read this slightly outdated article before -> #4450 , but it did not help me solving my issue.

I checked and changed my Gitea Configs (app.ini) with the Config-Sheet -> https://docs.gitea.io/en-us/config-cheat-sheet/ , hoping this could solve my problem. The changes are applied in my Configs, but all resources are still being used up.

Apart from this, the article above is about 2 years old and a few Gitea Versions were released since then, I would assume that this RAM/CPU/Resource problem should be fixed by now, since I am not the only one experiencing this issue.

I am grateful for all tips and hints.

If you need more detailed/further information, pls let me know.

Thank you all in advance!

...

Screenshots

[ashimokawa]

@JoeyWrk

This happens when migrating to 1.12 from a previous version, the language statistics generation causes this.
We faced the same problem, see here for discussion;

https://codeberg.org/Codeberg/Community/issues/198

[Ahaus314]

We have the same issue since v1.11.5. I thought to upgrate to 1.12.1 (at that time) would fix the issue. Our server is monitored by Prometheus/Grafana and we got warning stating the CPU is higher than 80% for more than five minutes… I receive this alert all day long. I tried to monitor the services, checked logs, but can't find what cause this issue. We are on Windows Server 2016.

If @ashimokawa cause is true, I hope the #11975 can fix this. I'm tired to plan a service restart after the business hours.

[zeripath]

You need to differentiate between load that occurs during migration and base load.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

[6543]

@JoeyWrk how many user and how many repos do this instance have?

[maxsnts]

Hi.
Going from 1.12.2 to 1.13.0 i faced the same problem.
Server unusable with all RAM (and swap) used.
Downgraded, problem went away.
Its not a huge instance, been working fine with 4GB, increased to 8GB but did not help.

Thanks

[zeripath]

Hi.
Going from 1.12.2 to 1.13.0 i faced the same problem.
Server unusable with all RAM (and swap) used.
Downgraded, problem went away.
Its not a huge instance, been working fine with 4GB, increased to 8GB but did not help.

Thanks

At the risk of repeating myself...

When did this happen?

Had gitea finished it's migration to 1.13 or was gitea up and running?

What caused the memory to spike? Was it a push? Are you pushing LFS or just normal files? How big?

Etc.

You're not giving us any information to help you.

[6543]

Its not a huge instance

1 repo, 10 repos 1000 ?

[maxsnts]

47 Repos. No one was doing any operation.
I'm running gitea as a daemon, maybe that is why +im not seeing the upgrade process?
Next time i will try running it the first time in the shell so that i can see the output.

[maxsnts]

Its something to do with many pulls at the same time.

We have a repo that gets checked periodically as part of a continuous delivery system.
Its about 150 VMs pulling a repo to see if there are new stuff.
Some times many of them happen to be at the same time.
In 1.12.5, these pulls go by with no problem at all.
In 1.13.0, gitea immediately exhausts all resources (RAM+CPU+SWAP) never recovering by it self.

Tested multiple times, the concurrent pulls is what kills gitea.

[zeripath]

OK so this is very likely related to the go-git reading objects in to memory - you might want to try the no-go-git PR. (#13673)

[maxsnts]

I'm sorry i'm a bit lost.
Thank you for the help, but when you can, can you point me to what it is "no-go-git PR"?
It may be obvious, but i'm not getting there.

[zeripath]

#13673

[maxsnts]

Got it.
I will try to get a build going.
Thanks.

[zeripath]

One other thing is that it might be helpful to set in your app.ini:

[server]
...
ENABLE_PPROF = true

Then use:

go tool pprof http://localhost:6060/debug/pprof/heap

and either type: top to get a top memory use and/or web to generate an svg of memory use.

Then at least we'll be able to trace where the memory is being used.

[dragospe]

Just wanted to chime in and say that I was having a similar issue that was fixed by building from #13673.
Thanks zeripath!

[maxsnts]

Great to hear, Thank you.

[techknowlogick]

Closing as linked PR is now merged.