Gitea leaves large uploads in /tmp

[s-hamann]

Description

We use Gitea with Drone. One of our CD jobs builds a large binary and uploads it to Gitea as a release (using Drone's Gitea-Release Plugin). This jobs runs nightly.
Apparently, Gitea stores the upload in /tmp and does not remove it from there. Therefore, the server runs out of available disk space after a while.

As a workaround, we manually restart Gitea every now and then. Due to Systemd's PrivateTmp this also wipes Giteas /tmp directory.

Note: We have

[repository.upload]
TEMP_PATH = data/tmp/uploads

So, I'm not sure, why uploads are stored in /tmp at all.

After one nightly job the file from the CD job is in Gitea's /tmp:

-rw------- 1 git git 221663149  3. Mai 02:23 multipart-940622690

To debug the issue, I experimented with uploading attachments to releases via the GUI and the API. In the browser, I'm limited to 20 MB file size and these files did not remain in /tmp.
When uploading the same 20 MB file with curl, it also did not remain in /tmp.
However, I could reproduce the issue by uploading a 30 MB file with curl.
So, the issue seems to be related to file size.
I can't tell, if only "large" files get written to /tmp or all files get written there, but only "small" files are deleted after the upload.

I did not investigate, if this issue affects only attachments to releases, or also other kinds of attachments.

It is unclear to me, what exactly the bug is, but I'm fairly certain it is a bug and not a configuration issue on our side.
I think Gitea should probably remove temporary uploads from /tmp or store them in TEMP_PATH instead of /tmp.

If you need any more information or want me to test something, please ask.

We'll update to the latest Gitea release by the end of the week. I'll report back if that changes anything.

Gitea Version

1.16.1

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

2.30.2

Operating System

Debian GNU/Linux 11.2

How are you running Gitea?

Gitea binary is from dl.gitea.io and started via a (custom) Systemd unit:

[Unit]
Description=Gitea (Git with a cup of tea)
After=network.target

[Service]
Type=simple
User=git
Group=git
WorkingDirectory=/var/lib/gitea
ExecStart=/opt/gitea/gitea web
Restart=always
Environment=USER=git HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea GITEA_CUSTOM=/opt/gitea/custom MACARON_ENV=production
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true

[Install]
WantedBy=multi-user.target

Database

PostgreSQL

[wxiaoguang]

I haven't looked into the problem. Now I can tell some information:

1. The answer to the size is 32M
2. It's caused by Golang's mime/multipart/formdata.go, if a form is too large(32M), it will store the file into a temp file by
   file, err := os.CreateTemp("", "multipart-")
3. I have no idea why the temp file is not deleted, if anyone has time to take a look and has clues, please share or fix.

[wxiaoguang]

I have a feeling that maybe this problem is upstream related ....

• mime/multipart: TempFile file hangs around on disk after usage in multipart/formdata.go golang/go#20253

Or .... Gitea does something wrong ....

[jolheiser]

As noted in that ticket, perhaps we aren't using https://pkg.go.dev/mime/multipart#Form.RemoveAll
As a hint to anyone who looks

[wxiaoguang]

It should be called by Golang's http framework automatically. Not sure why it doesnt' work correctly.

in server.go

func (w *response) finishRequest() {
	.....
	if w.req.MultipartForm != nil {
		w.req.MultipartForm.RemoveAll()
	}
}

[wxiaoguang]

I think I find the possible root case.

Gitea's ctx.Req is a clone of Golang http framework's request, so the ctx.Req.MultipartForm is not managed by Golang.

Just a guess.

[s-hamann]

1. The answer to the size is `32M`

Just to be precise: I could reproduce the issue with a file of exactly 31457280 bytes, which is less than either 32 MB or 32 MiB. Not sure if this is really relevant here.

[wxiaoguang]

Could you show your curl command? I could try to fix it if I can confirm the problem.

[s-hamann]

Could you show your curl command? I could try to fix it if I can confirm the problem.

I had Swagger generate the curl command. This is the result (with some placeholders):

curl -X 'POST' 'https://$domain/api/v1/repos/$user/$repo/releases/$id/assets?token=$api_token' -H 'accept: application/json' -H 'Content-Type: multipart/form-data' -F 'attachment=@$file'

The release I uploaded to was manually created using Gitea's GUI.

And here's how I generated my test file:

dd if=/dev/zero of=$file bs=1024 count=30

[wxiaoguang]

OK, everything is clear.

My fix works as expected. No more temp files in /tmp after the patch.

The answer to 32M is: Gitea overwrites Golang's maxMemory by:

if err := ctx.Req.ParseMultipartForm(setting.Attachment.MaxSize << 20); err != nil && !strings.Contains(err.Error(), "EOF") 

So if your file exceeds setting.Attachment.MaxSize << 20 (in app.ini), then the post body will be buffered on to disk.