Description
- Gitea version (or commit ref): 1.10.0
- Git version: 2.16.4
- Operating system: openSUSE Leap 15.1
- Database (use
[x]
):- PostgreSQLMySQL (MariaDB)MSSQLSQLite
- Can you reproduce the bug at https://try.gitea.io:
- Yes (provide example URL)NoNot relevant
- Log gist:
Description
As written earlier in the Discourse forum, it regularly happens that gitea makes my server's iowait and memory consumption skyrocket, causing issues for all other services running within the same VM. Usually only killing the whole gitea service makes the server recover.
While I couldn't find any direct cause by parsing the logs or the PPROF data, I just discovered in the admin web interface (tab Monitoring) that the GetDiffRange operation seems to be responsible. It seems to get stuck at a single repo of just 138MiB size, clogging IO and making gitea consume more and more memory. iotop
shows that gitea creates a lot of I/O, although I'm not sure whether swapping operations are allocated to the process causing them as well.
After this issue first occured, I limited the gitea systemd service to 600MiB and gitea is currently using all of it. But apparently this limit isn't working as the memory+swap usage increased by more than 3GiB during the GetDiffRange operation.
What does this GetDiffRange operation do? I wasn't able to find information on that.
Luckily I was able to allocate an additional 1GiB of memory to my VM, giving me the chance to let the operation run. So far it's running for more than 30 minutes though.
Any ideas about that operation, why it's getting stuck or any additional debug information I could provide?
Activity
schmittlauch commentedon Dec 6, 2019
After ~45min the GetDiffRange operation vanished from the Monitoring web UI, but both memory consumption and IOwait remain equally high. I see myself forced to kill gitea after additional 15 minutes.
schmittlauch commentedon Dec 6, 2019
Killing the gitea service immediately freed up 4.5GiB of memory.
lunny commentedon Dec 8, 2019
There is a bug about
OpenRepository
has been fixed on v1.9.6 or v1.10.0(1?). Could you try v1.10.1?schmittlauch commentedon Dec 8, 2019
will do
zeripath commentedon Dec 9, 2019
I think it's unlikely to be fixed by that - the architecture of GetDiffRange keeps the whole diff in memory. (Probably multiple copies of it too.) If you have a huge diff you're going to break Gitea.
typeless commentedon Dec 12, 2019
References:
https://github.blog/2016-12-06-how-we-made-diff-pages-3x-faster/
https://github.com/go-gitea/gitea/blob/master/services/pull/merge.go#L504-L545
lunny commentedon Dec 12, 2019
We can also cache the diff sections to cache server or files.
[-]GetDiffRange operation causes iowait and memory consumption to skyrocket[/-][+]iowait and memory consumption are skyrocketing regularly[/+]schmittlauch commentedon Dec 13, 2019
The issue is still happening regularly even with v1.10.1, but I haven't always seen the "GetDiffRange" operation showing up on the Monitoring page. So GetDiffRange isn't necessarily the cause of this issue.
The gitea logs only show a huge amount of mysql queries and I wasn't able to get anything from the PROF data (I'm not a go developer).
If anyone thinks they can get relevant information from there, I'm willing to pass you the logs and prof data.
lunny commentedon Dec 14, 2019
So that maybe some cron tasks.
zeripath commentedon Dec 14, 2019
Hmm excessive Perm checks are a possible cause - see the comments on #8540?
There's another possible issue - testPatch was previously reading the whole patch in to memory. That should stop now.
schmittlauch commentedon Dec 14, 2019
zeripath commentedon Dec 14, 2019
Changes to PRs would cause that
zeripath commentedon Dec 14, 2019
Do you have repo indexer on? Maybe that could be a cause?
schmittlauch commentedon Dec 16, 2019
I didn't modify any PRs in the meantime and do not have the indexer enabled (if it's disabled by default). git-lfs is enabled though.
Nevertheless after a while my gitea uses 2GiB of memory:
Server Uptime
9 hours, 34 minutes, 53 seconds
Current Goroutines
29
Current Memory Usage
44MB
Total Memory Allocated
17GB
Memory Obtained
2.8GB
Pointer Lookup Times
0
Memory Allocations
103340464
Memory Frees
102837346
Current Heap Usage
44MB
Heap Memory Obtained
2.7GB
Heap Memory Idle
2.6GB
Heap Memory In Use
52MB
Heap Memory Released
2.6GB
Heap Objects
503118
Bootstrap Stack Usage
928KB
Stack Memory Obtained
928KB
MSpan Structures Usage
678KB
MSpan Structures Obtained
8.2MB
MCache Structures Usage
5.1KB
MCache Structures Obtained
16KB
Profiling Bucket Hash Table Obtained
2.3MB
GC Metadata Obtained
97MB
Other System Allocation Obtained
1.6MB
Next GC Recycle
86MB
Since Last GC Time
19.3s
Total GC Pause
0.2s
Last GC Pause
0.000s
GC Times
382
(taken from the admin dashboard)
25 remaining items
eikendev commentedon Jul 28, 2020
It was a repository where a file of over 100MB was added, something that actually belongs into lfs.
I guess my specific case can be detected by looking for large files inside the repository. About the general case, maybe
git log --stat
helps.zeripath commentedon Jul 28, 2020
@eikendev that's bad - do you know if it was when the file was pushed or when someone looked at the diff page?
zeripath commentedon Jul 28, 2020
oo interesting I just pushed a file with no newlines and memory shot up.
eikendev commentedon Jul 28, 2020
In the graph I posted earlier I have neither pushed anything nor loaded any huge diff. I looked at the repo at some points, but not the diffs. When memory is freed in the graph, this would indicate I manually restarted the server.
To me this seems more like a product of a scheduled task.
I'm not sure right now if my issue and @schmittlauch's are related.
lafriks commentedon Jul 28, 2020
Do you see what is exact git command (with arguments) that is left dangling?
schmittlauch commentedon Jul 28, 2020
@lafriks Currently there is no dangling git command after having restarted gitea.
I'll keep that in mind for the future, but the growth of memory consumption happens independent of dangling git processes being present or not. Right now my 3.5GiB memory are nearly full, the growth will continue into swap the next day.
zeripath commentedon Feb 14, 2021
OK I think this is going to be related to go-git related issues.
I think therefore I'm going to pull the confirmed from this tag to see if 1.14 still has this issue.
zeripath commentedon Dec 15, 2021
I think I might have discovered the underlying issue here - if git cat-file is called on a broken git repository it will hang until stdin is closed instead of fatalling immediately. I think therefore #17991 will fix this.
schmittlauch commentedon Dec 15, 2021
This issue might even be resolved already, but I cannot say for sure as I moved my installation to a new installation, cleaned up some old repos, and thus might have removed such a broken repo coincidentally.
So I suggest to indeed close this once #17991 gets merged, unless some new instance of these symptoms here comes up.
Prevent hang in git cat-file if repository is not a valid repository …
Prevent hang in git cat-file if repository is not a valid repository …