slow git blame on files with large history leading to OOM kills

[akosyakov]

Bug description

Since last 2 weeks I start getting OOM kills while working on https://github.com/gitpod-io/vscode repo. After investigation it turned out that doing some git operations causes git gc/repack which runs for long sometimes and consumes memory. So newly started processes cannot get more memory and gets killed, eventually leading to the workspace eviction. I found several triggers in my settings: (1) i had auto enabled gti fetch, (2) git blame via gitlens, as wells as (3) git push causes git gc from time to time. In particular blaming via GitLens can get unresponsive for minutes. You can see it on the screenshot below:

Don't pay attention to htop saying 33.G/62.8G. It does not know about gcroups. Instead have a look at the status bar, it shows almost 100% utilisation of memory limited by cgroups, which is used to trigger OOM kills. It is computed here: https://github.com/akosyakov/gitpod-monitor/blob/96d342a3abb13e6608dbd48fdf9efbc4c0700b55/src/extension.ts#L37-L39

I suspect a cause for it is the partial cloning.

Steps to reproduce

Install GitLens and use blame on different files in https://github.com/gitpod-io/vscode while running some task terminals. My particular trigger was doing yarn gitpod-min.

Expected behavior

No response

Example repository

No response

Anything else?

No response

[csweichel]

What happens if you do the same thing on a repo cloned without --filter?

[svenefftinge]

I can reproduce a super slow git blame.
E.g.

git blame src/vs/workbench/contrib/tasks/browser/abstractTaskService.ts

Takes minutes. I wonder if git has some sort of inflate operation, that would allow us to fetch the blobs async after the content initialization.

[csweichel]

I wonder if git has some sort of inflate operation, that would allow us to fetch the blobs async after the content initialization.

been thinking the same thing - had a look a few days ago while looking into the origin issue, but found nothing. That's not to say such an operation doesn't exist, but it doesn't seem to be obvious. If we had such a thing though, we could heuristically inflate things.

Also, we should double check the Git issues. Is this resource demand/bad performance a known issue?

[svenefftinge]

Edit: This doesn't do the job!

Just stumbled over

git -c "remote.origin.partialclonefilter=blob:none" fetch --deepen=999999 origin

which seems to do the trick. Source

[csweichel]

it does not help when calling git blame unfortunately. To my understanding the command above helps when cloning the repo with a tree filter, rather than a blob filter.
Removing the blob filter altogether doesn't help either: git -c "remote.origin.partialclonefilter=" fetch --deepen=999999 origin

[svenefftinge]

True, I ran it on a repo that was already inflated by running git blame on a large file, so blame on other files were instant afterward.

Generally, the inflation wouldn't help us much if we were running it eagerly, because many users would not wait for git clone but for content init from a prebuild. So if we always inflate it we are back to the same size on disk.

[akosyakov]

Discussed on the engineering call that to add it to the groundwork inbox to research on the solution which can preserve partial cloning and inflate the workspace afterwards when it is needed. Maybe add a flag in .gipod.yml to enable such inflation for special cases, like this one and #4914

[aledbf]

Maybe we can make this git feature optional using a setting to the gitpod.yml file?

[JanKoehnlein]

Let's investigate this with a timebox of one day.

/schedule

[svenefftinge]

I think that we should make the partial clone an opt-in feature that one can enable through .gitpod.yml.

It's too bad this improvement is not transparent enough to make it opt-out, but the impact of not being able to fork (#4914) and sometimes experiencing oom kills and/or long waiting times on git blame is surprising and will leave users with a bad experience. I'd rather compromise on the loading speed than the functional experience.

[akosyakov]

VS Code (git) does not allow to commit sometimes because of it. Actually it allows but it shows this dialog each time.