Git LFS Push 500 Internal Server Error

[Shinmera]

Description

When attempting to push new LFS artefacts, the internal logs only show error messages like so, with the last error repeating over and over:

Apr 11 13:10:40 imai gitea[556860]: 2023/04/11 13:10:40 [64354030] router: completed GET /api/internal/serv/command/10/shinmera/trial-deployment?mode=2&verb=git-receive-pack for 31.164.105.196:0, 200 OK in 2.4ms @ private/serv.go:81(private.ServCommand)
Apr 11 13:10:41 imai gitea[556860]: 2023/04/11 13:10:41 [64354031] router: completed POST /api/internal/ssh/log for 31.164.105.196:0, 200 OK in 0.1ms @ private/ssh_log.go:18(private.SSHLog)
Apr 11 13:10:41 imai gitea[556860]: 2023/04/11 13:10:41 [64354031-2] router: completed GET /api/internal/serv/command/10/shinmera/trial-deployment?mode=1&verb=git-upload-pack for 31.164.105.196:0, 200 OK in 2.1ms @ private/serv.go:81(private.ServCommand)
Apr 11 13:10:42 imai gitea[556860]: 2023/04/11 13:10:42 [64354031-3] router: completed POST /api/internal/ssh/10/update/106 for 31.164.105.196:0, 200 OK in 129.7ms @ private/key.go:18(private.UpdatePublicKeyInRepo)
Apr 11 13:10:42 imai gitea[556860]: 2023/04/11 13:10:42 [64354032] router: completed POST /api/internal/ssh/log for 31.164.105.196:0, 200 OK in 0.1ms @ private/ssh_log.go:18(private.SSHLog)
Apr 11 13:10:42 imai gitea[556860]: 2023/04/11 13:10:42 [64354032-2] router: completed GET /api/internal/serv/command/10/shinmera/trial-deployment?mode=2&verb=git-lfs-authenticate&verb=upload for 31.164.105.196:0, 200 OK in 2.7ms @ private/serv.go:81(private.ServCommand)
Apr 11 13:10:43 imai gitea[556860]: 2023/04/11 13:10:43 [64354033] router: completed POST /shinmera/trial-deployment.git/info/lfs/locks/verify for 10.1.2.4:50662, 200 OK in 2.6ms @ lfs/locks.go:200(lfs.VerifyLockHandler)
Apr 11 13:10:43 imai gitea[556860]: 2023/04/11 13:10:43 [64354033-2] router: completed POST /shinmera/trial-deployment.git/info/lfs/objects/batch for 10.1.2.4:50664, 200 OK in 25.2ms @ lfs/server.go:154(lfs.BatchHandler)
Apr 11 13:10:44 imai gitea[556860]: 2023/04/11 13:10:44 [64354034-18] router: completed PUT /shinmera/trial-deployment.git/info/lfs/objects/fc663907272b18264b5e35c8d428aa378948d57a532f01452287d6fcdfad463c/49906 for 10.1.2.4:50968, 500 Internal Server Error in 10.0ms @ lfs/server.go:270(lfs.UploadHandler)

The logs contain nothing more useful, even when run in dev mode.

Pushing non-LFS content works just fine. I checked the file access permissions, everything is owned by the group and user git, and gitea is running under that user and group. I also tried setting the safe.directory git config setting to * or to include all relevant gitea storage directories, without success.

Gitea version 1.18.5 built with GNU Make 4.4.1, go1.20.1 : bindata, sqlite, sqlite_unlock_notify, pam

app.ini, passwords etc replaced by ...:

APP_NAME = TyNET Gitea
RUN_USER = git
RUN_MODE = prod

[security]
INTERNAL_TOKEN = ...
INSTALL_LOCK   = true
SECRET_KEY     = ...

[server]
LOCAL_ROOT_URL   = http://localhost:80/
SSH_DOMAIN       = git.tymoon.eu
DOMAIN           = localhost
HTTP_PORT        = 80
HTTP_ADDR        = 0.0.0.0
ROOT_URL         = https://gitea.tymoon.eu/
DISABLE_SSH      = false
SSH_PORT         = 10422
LFS_START_SERVER = true
LFS_JWT_SECRET   = ...

[lfs]
PATH = /var/lib/gitea/data/lfs

[database]
DB_TYPE  = postgres
HOST     = ...
NAME     = gitea
USER     = gitea
PASSWD   = ...
SSL_MODE = disable

[repository]
ROOT = /var/lib/gitea/repos

[service]
REGISTER_EMAIL_CONFIRM            = false
ENABLE_NOTIFY_MAIL                = true
DISABLE_REGISTRATION              = true
ALLOW_ONLY_EXTERNAL_REGISTRATION  = false
ENABLE_CAPTCHA                    = false
REQUIRE_SIGNIN_VIEW               = false
DEFAULT_KEEP_EMAIL_PRIVATE        = false
DEFAULT_ALLOW_CREATE_ORGANIZATION = true
DEFAULT_ENABLE_TIMETRACKING       = true
NO_REPLY_ADDRESS                  = noreply.example.org

[picture]
DISABLE_GRAVATAR        = false
ENABLE_FEDERATED_AVATAR = true

[openid]
ENABLE_OPENID_SIGNIN = false
ENABLE_OPENID_SIGNUP = false

[session]
PROVIDER = file

[log]
MODE      = file
LEVEL     = Debug
ROOT_PATH = /var/lib/gitea/log

[log.sublogger.macaron]
LEVEL = WARN

[ui]
DEFAULT_THEME = arc-green

[repository.upload]
ALLOWED_TYPES =
FILE_MAX_SIZE = 1000
MAX_FILES     = 100

[attachment]
ALLOWED_TYPES = image/jpeg|image/png|application/zip|application/gzip|application/x-xz|application/zlib|application/x-dosexec|application/octet-stream|application/x-msdownload|application/vnd.microsoft.portable-executable|application/x-apple-diskimage|application/x-compressed-tar|application/exe|application/x-exe|application/dos-exeapplication/x-winexe|application/msdos-windows|application/x-msdos-program|application/pdf
MAX_SIZE      = 1000
MAX_FILES     = 100

[oauth2]
JWT_SECRET = ...

Gitea Version

Gitea version 1.18.5 built with GNU Make 4.4.1, go1.20.1 : bindata, sqlite, sqlite_unlock_notify, pam

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

git version 2.40.0

Operating System

Linux imai 6.2.6-arch1-1 #1 SMP PREEMPT_DYNAMIC Mon, 13 Mar 2023 17:02:08 +0000 x86_64 GNU/Linux

How are you running Gitea?

Using official packages from the Arch Linux repositories, run as a systemd service.

Database

PostgreSQL

[sillyguodong]

Are there any associated errors in gitea.log?

[Shinmera]

Not as far as I can see. I've uploaded the corresponding timestamp from the log:

https://gist.github.com/Shinmera/13f31bf25a27da09e7a82cbff7cdeb38

[wxiaoguang]

The most suspicious part is that there are a lot of ROLLBACK commands in the SQL log.

And it explains that why there is no error log, because at the moment, the code is, the database failure doesn't print the error .....

Do you know if there is anything wrong with the lfs_meta_object table?

[Shinmera]

Ah, good shout!

I had a peek around with pgadmin, and it seems the lfs_meta_object sequence somehow got out of sync, possibly due to a bad backup job at some point? Fixing the sequence resolved the issue.

Thanks a lot for the debugging help, and my apologies for the noise!

[wxiaoguang]

Never mind, I also find that the log is pretty unfriendly .... so I propose #24072 😁