Skip to content

iowait and memory consumption are skyrocketing regularly #9271

Closed
@schmittlauch

Description

@schmittlauch
  • Gitea version (or commit ref): 1.10.0
  • Git version: 2.16.4
  • Operating system: openSUSE Leap 15.1
  • Database (use [x]):
    • PostgreSQL
      MySQL (MariaDB)
      MSSQL
      SQLite
  • Can you reproduce the bug at https://try.gitea.io:
    • Yes (provide example URL)
      No
      Not relevant
  • Log gist:

Description

As written earlier in the Discourse forum, it regularly happens that gitea makes my server's iowait and memory consumption skyrocket, causing issues for all other services running within the same VM. Usually only killing the whole gitea service makes the server recover.
While I couldn't find any direct cause by parsing the logs or the PPROF data, I just discovered in the admin web interface (tab Monitoring) that the GetDiffRange operation seems to be responsible. It seems to get stuck at a single repo of just 138MiB size, clogging IO and making gitea consume more and more memory. iotop shows that gitea creates a lot of I/O, although I'm not sure whether swapping operations are allocated to the process causing them as well.
After this issue first occured, I limited the gitea systemd service to 600MiB and gitea is currently using all of it. But apparently this limit isn't working as the memory+swap usage increased by more than 3GiB during the GetDiffRange operation.

What does this GetDiffRange operation do? I wasn't able to find information on that.

Luckily I was able to allocate an additional 1GiB of memory to my VM, giving me the chance to let the operation run. So far it's running for more than 30 minutes though.

Any ideas about that operation, why it's getting stuck or any additional debug information I could provide?

Activity

schmittlauch

schmittlauch commented on Dec 6, 2019

@schmittlauch
ContributorAuthor

After ~45min the GetDiffRange operation vanished from the Monitoring web UI, but both memory consumption and IOwait remain equally high. I see myself forced to kill gitea after additional 15 minutes.

schmittlauch

schmittlauch commented on Dec 6, 2019

@schmittlauch
ContributorAuthor

Killing the gitea service immediately freed up 4.5GiB of memory.

lunny

lunny commented on Dec 8, 2019

@lunny
Member

There is a bug about OpenRepository has been fixed on v1.9.6 or v1.10.0(1?). Could you try v1.10.1?

schmittlauch

schmittlauch commented on Dec 8, 2019

@schmittlauch
ContributorAuthor

will do

zeripath

zeripath commented on Dec 9, 2019

@zeripath
Contributor

I think it's unlikely to be fixed by that - the architecture of GetDiffRange keeps the whole diff in memory. (Probably multiple copies of it too.) If you have a huge diff you're going to break Gitea.

lunny

lunny commented on Dec 12, 2019

@lunny
Member

We can also cache the diff sections to cache server or files.

changed the title [-]GetDiffRange operation causes iowait and memory consumption to skyrocket[/-] [+]iowait and memory consumption are skyrocketing regularly[/+] on Dec 13, 2019
schmittlauch

schmittlauch commented on Dec 13, 2019

@schmittlauch
ContributorAuthor

The issue is still happening regularly even with v1.10.1, but I haven't always seen the "GetDiffRange" operation showing up on the Monitoring page. So GetDiffRange isn't necessarily the cause of this issue.

The gitea logs only show a huge amount of mysql queries and I wasn't able to get anything from the PROF data (I'm not a go developer).
If anyone thinks they can get relevant information from there, I'm willing to pass you the logs and prof data.

lunny

lunny commented on Dec 14, 2019

@lunny
Member

So that maybe some cron tasks.

zeripath

zeripath commented on Dec 14, 2019

@zeripath
Contributor

Hmm excessive Perm checks are a possible cause - see the comments on #8540?

There's another possible issue - testPatch was previously reading the whole patch in to memory. That should stop now.

schmittlauch

schmittlauch commented on Dec 14, 2019

@schmittlauch
ContributorAuthor
zeripath

zeripath commented on Dec 14, 2019

@zeripath
Contributor

Changes to PRs would cause that

zeripath

zeripath commented on Dec 14, 2019

@zeripath
Contributor

Do you have repo indexer on? Maybe that could be a cause?

schmittlauch

schmittlauch commented on Dec 16, 2019

@schmittlauch
ContributorAuthor

I didn't modify any PRs in the meantime and do not have the indexer enabled (if it's disabled by default). git-lfs is enabled though.
Nevertheless after a while my gitea uses 2GiB of memory:

Server Uptime
9 hours, 34 minutes, 53 seconds
Current Goroutines
29
Current Memory Usage
44MB
Total Memory Allocated
17GB
Memory Obtained
2.8GB
Pointer Lookup Times
0
Memory Allocations
103340464
Memory Frees
102837346
Current Heap Usage
44MB
Heap Memory Obtained
2.7GB
Heap Memory Idle
2.6GB
Heap Memory In Use
52MB
Heap Memory Released
2.6GB
Heap Objects
503118
Bootstrap Stack Usage
928KB
Stack Memory Obtained
928KB
MSpan Structures Usage
678KB
MSpan Structures Obtained
8.2MB
MCache Structures Usage
5.1KB
MCache Structures Obtained
16KB
Profiling Bucket Hash Table Obtained
2.3MB
GC Metadata Obtained
97MB
Other System Allocation Obtained
1.6MB
Next GC Recycle
86MB
Since Last GC Time
19.3s
Total GC Pause
0.2s
Last GC Pause
0.000s
GC Times
382

(taken from the admin dashboard)

25 remaining items

eikendev

eikendev commented on Jul 28, 2020

@eikendev

@eikendev What do you categorise as huge diffs and how can I check whether one of my repos falls into that category?

It was a repository where a file of over 100MB was added, something that actually belongs into lfs.

I guess my specific case can be detected by looking for large files inside the repository. About the general case, maybe git log --stat helps.

zeripath

zeripath commented on Jul 28, 2020

@zeripath
Contributor

@eikendev that's bad - do you know if it was when the file was pushed or when someone looked at the diff page?

zeripath

zeripath commented on Jul 28, 2020

@zeripath
Contributor

oo interesting I just pushed a file with no newlines and memory shot up.

eikendev

eikendev commented on Jul 28, 2020

@eikendev

@eikendev that's bad - do you know if it was when the file was pushed or when someone looked at the diff page?

In the graph I posted earlier I have neither pushed anything nor loaded any huge diff. I looked at the repo at some points, but not the diffs. When memory is freed in the graph, this would indicate I manually restarted the server.

To me this seems more like a product of a scheduled task.

I'm not sure right now if my issue and @schmittlauch's are related.

lafriks

lafriks commented on Jul 28, 2020

@lafriks
Member

Do you see what is exact git command (with arguments) that is left dangling?

schmittlauch

schmittlauch commented on Jul 28, 2020

@schmittlauch
ContributorAuthor

@lafriks Currently there is no dangling git command after having restarted gitea.
I'll keep that in mind for the future, but the growth of memory consumption happens independent of dangling git processes being present or not. Right now my 3.5GiB memory are nearly full, the growth will continue into swap the next day.

zeripath

zeripath commented on Feb 14, 2021

@zeripath
Contributor

OK I think this is going to be related to go-git related issues.

I think therefore I'm going to pull the confirmed from this tag to see if 1.14 still has this issue.

removed
issue/confirmedIssue has been reviewed and confirmed to be present or accepted to be implemented
on Feb 14, 2021
zeripath

zeripath commented on Dec 15, 2021

@zeripath
Contributor

I think I might have discovered the underlying issue here - if git cat-file is called on a broken git repository it will hang until stdin is closed instead of fatalling immediately. I think therefore #17991 will fix this.

schmittlauch

schmittlauch commented on Dec 15, 2021

@schmittlauch
ContributorAuthor

This issue might even be resolved already, but I cannot say for sure as I moved my installation to a new installation, cleaned up some old repos, and thus might have removed such a broken repo coincidentally.

So I suggest to indeed close this once #17991 gets merged, unless some new instance of these symptoms here comes up.

added a commit that references this issue on Dec 16, 2021
8354670
added a commit that references this issue on Mar 28, 2022
c7c8e47
locked and limited conversation to collaborators on Apr 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performance/memoryPerformance issues affecting memory useperformance/speedperformance issues with slow downs

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Participants

      @lunny@lafriks@schmittlauch@zeripath@typeless

      Issue actions

        iowait and memory consumption are skyrocketing regularly · Issue #9271 · go-gitea/gitea