Search code, repositories, users, issues, pull requests...

opened

on Nov 18, 2021

Contributor

Gitea Version

1.15.6

Git Version

Seen on different servers running 2.25.1 and 2.34.0

Operating System

Ubuntu 20.04.3

How are you running Gitea?

One server is manually deployed from a GitHub release and started with a systemd script. The other uses the apt-installed version of Gitea.

Database

PostgreSQL

Can you reproduce the bug on the Gitea demo site?

Log Gist

I can gather a log if that would be helpful. I've included clear repro steps in the description.

Description

LFS files are failing to download from forks. The only reason I can't repro on try.gitea.io is because LFS doesn't seem to be enabled. This is my minimal repro (visible at https://git.pernicious.games/try-gitea-lfs/lfs-fork-test and the fork at https://git.pernicious.games/cpickett/lfs-fork-test/src/branch/fork-1).

Note that we are pretty sure this used to work on our Gitea installation months ago, so it may be caused by a relatively recent Gitea version (maybe 1.15+).

The repro steps are:

Create new repo with an LFS file
Fork that repo
Commit a new rev of the LFS file in a branch on the fork
Open a PR for the main repo from the fork's branch

Now clone the base repo somewhere else and try to either pull from the remote or the PR ref:

Clone base repo (e.g. git clone https://git.pernicious.games/try-gitea-lfs/lfs-fork-test.git)
Add the fork as a remote and fetch it (e.g.: git remote add parnic https://git.pernicious.games/cpickett/lfs-fork-test.git, git fetch parnic)
Try switching to the branch that exists in the fork with the modified LFS file (e.g.: git switch fork-1)
- Receive error, e.g.:

Downloading test.lfs (9 B)
Error downloading object: test.lfs (7ffc8bb): Smudge error: Error downloading test.lfs (7ffc8bb3b0c443a3d170b1f6aa402a66c6de86c49dd2d74e452d6d82b855939f): [7ffc8bb3b0c443a3d170b1f6aa402a66c6de86c49dd2d74e452d6d82b855939f] Not Found: [404] Not Found

Restore modified files and try switching to the PR ref instead (e.g.: git restore ., git fetch origin pull/1/head:test, git switch test)
- Receive same error as above

Note that the LFS file at the specified revision/hash exists in the fork, e.g.: https://git.pernicious.games/cpickett/lfs-fork-test.git/info/lfs/objects/7ffc8bb3b0c443a3d170b1f6aa402a66c6de86c49dd2d74e452d6d82b855939f/direct
but it does not exist in the base repo: https://git.pernicious.games/try-gitea-lfs/lfs-fork-test.git/info/lfs/objects/7ffc8bb3b0c443a3d170b1f6aa402a66c6de86c49dd2d74e452d6d82b855939f/direct

This is important because when running git lfs fetch --all we see that it is trying to pull from the base org, not the fork:

>git lfs fetch --all
fetch: 2 object(s) found, done.
fetch: Fetching all references...
[7ffc8bb3b0c443a3d170b1f6aa402a66c6de86c49dd2d74e452d6d82b855939f] Not Found: [404] Not Found
error: failed to fetch some objects from 'https://git.pernicious.games/try-gitea-lfs/lfs-fork-test.git/info/lfs'

Finally, note that the LFS file does exist on the disk under the generic LFS directory. For this example, that's
/var/lib/gitea/data/lfs/7f/fc/8bb3b0c443a3d170b1f6aa402a66c6de86c49dd2d74e452d6d82b855939f

Screenshots

No response

wxiaoguang

Contributor

Can @zeripath or @KN4CK3R or someone else help to confirm?

IIRC the LFS code gets some changes during 1.14 -> 1.15, I am not sure whether this is a permission problem or a database consistency problem.

Member

Are you sure that worked before? After creating the PR the base repository contains the dangling pointer without the database reference:

(top = base, bottom = pr branch)

During 1.14 -> 1.15 I changed the LFS API code but this looks like we need to sync the LFS objects when creating a PR. And that functionality was never implemented I think.

Member

Looks like, I was wrong. I tested 1.12 and it works with that version (but is not correct I think). After creating the PR blob list does only show a single pointer (that's wrong because of the new PR branch?). Seems like the download request associated the pointer, so my changes in #16865 or #15523 may be the reason of the problem.

Member

The old code created the database references when downloading files. I changed that in #15523 so that only upload requests are able to do that.

Reasons:

The download request should just not do that (?)
To associate the file only Read-rights were enforced
The user could associate every file from different repositories (private too) to this repo if he knows the pointer data (was fixed in Test if LFS object is accessible #16865)

So the question is how we would like to handle this problem. Personally I don't like to bring back file association in the download request. Auto-sync when creating a PR?

added

Member

The old code created the database references when downloading files. I changed that in #15523 so that only upload requests are able to do that.

Reasons:

The download request should just not do that (?)

To associate the file only Read-rights were enforced

The user could associate every file from different repositories (private too) to this repo if he knows the pointer data (was fixed in Test if LFS object is accessible #16865)

So the question is how we would like to handle this problem. Personally I don't like to bring back file association in the download request. Auto-sync when creating a PR?

I vote later.

wxiaoguang

Contributor

And just FYI, I remember this issue: #17207 , are they related? That issue also said about the behavior differs between Gitea release.

Member

Yes, I think they are related. Just putting the files in the directory does not associate the repo to them.

stu1811

I agree they are probably related and I've run into this issue as well. As a workaround you can do a git lfs fetch <UPSTREAM> and then a git lfs checkout to get the files.

parnic-sks

added a commit that references this issue

on Nov 22, 2021

Associate LFS pointer on download

768aaab

ContributorAuthor

We are hosting a private Gitea server behind a VPN, so I've temporarily reintroduced the associate-LFS-pointers-on-download code for a hacked-up local build, in case anyone else needs the temporary workaround: https://github.com/go-gitea/gitea/compare/v1.15.6...parnic-sks:associate-lfs-obj-on-download?expand=1

stu1811

And just FYI, I remember this issue: #17207 , are they related? That issue also said about the behavior differs between Gitea release.

I built 1.15.6 cherry-picked the changes from parnic-sks@768aaab and it did not fix issue #17207. Detail below.

We got a new bundle with LFS updates. I extracted the new LFS files, fetched the branch, and tried to push to origin. It would not push due hint: Your push was rejected due to missing or corrupt local objects. I also tried git lfs fetch --all and the push failed again.

ContributorAuthor

And just FYI, I remember this issue: #17207 , are they related? That issue also said about the behavior differs between Gitea release.

I built 1.15.6 cherry-picked the changes from parnic-sks@768aaab and it did not fix issue #17207. Detail below.

We got a new bundle with LFS updates. I extracted the new LFS files, fetched the branch, and tried to push to origin. It would not push due hint: Your push was rejected due to missing or corrupt local objects. I also tried git lfs fetch --all and the push failed again.

That one commit is not enough, you need the entire branch diff. parnic-sks@3872374 was actually the correct fix, but it needs the commit after that to fix the nil user deref.

stu1811

And just FYI, I remember this issue: #17207 , are they related? That issue also said about the behavior differs between Gitea release.

I built 1.15.6 cherry-picked the changes from parnic-sks@768aaab and it did not fix issue #17207. Detail below.
We got a new bundle with LFS updates. I extracted the new LFS files, fetched the branch, and tried to push to origin. It would not push due hint: Your push was rejected due to missing or corrupt local objects. I also tried git lfs fetch --all and the push failed again.

That one commit is not enough, you need the entire branch diff. parnic-sks@3872374 was actually the correct fix, but it needs the commit after that to fix the nil user deref.

Sorry I mean I cherry-picked the whole branch.

wxiaoguang

mentioned this

on Dec 31, 2021

Allow admin to associate missing LFS objects for repositories #18143

54 remaining items

test-sha256

So the question is how we would like to handle this problem. Personally I don't like to bring back file association in the download request. Auto-sync when creating a PR?

While I agree that associate on download is not ideal, there should be an auto-sync that associates all the files in PR. This would allow the commits and its LFS objects can be checked out in target repository.

gamedevkirk

Hoping this gets some attention at some point. This is a brutally frustrating issue.

gitea/routers/web/repo/setting/lfs.go

mentioned this

on Dec 3, 2024

LFS files missing in PR openSUSE/openSUSE-git#72

StanleySweet

Contributor

I believe 0 A.D. has the same, or a similar issue when syncing forks via the web UI.

na-Itms

Contributor

There are many situations in which our contributors for 0 A.D. (artists and programmers alike) run into the issue. We are forced to apply the workaround linked above.

In the specific situation of the "Sync Fork" button, a contributor updates their main branch using the "Sync Fork" button, which fast-forwards their main branch to the latest upstream. However, LFS files are not associated in the fork. Thus, the fork is not cloneable (without the workaround, it 404s on cloning).

I think that a proper fix for this specific situation would be to run the settings/lfs/pointers/associate endpoint every time the "Sync Fork" button is used.

Line 515 in e9f5105

func LFSAutoAssociate(ctx *context.Context) {

I have tried to implement this and send you a PR, but I am not a Go developer and I don't understand how to properly call a method under routers/settings from a routers/branch context.

Member

Add a trace for the last step.

$ GIT_TRACE=1 GIT_LFS_TRACE=1 git sw -C fork-1 fork/fork-1
21:25:01.076910 git.c:772               trace: exec: git-sw -C fork-1 fork/fork-1
21:25:01.077147 run-command.c:667       trace: run_command: git-sw -C fork-1 fork/fork-1
21:25:01.078286 git.c:423               trace: alias expansion: sw => switch
21:25:01.078298 git.c:830               trace: exec: git switch -C fork-1 fork/fork-1
21:25:01.078301 run-command.c:667       trace: run_command: git switch -C fork-1 fork/fork-1
21:25:01.078310 run-command.c:759       trace: start_command: /opt/homebrew/opt/git/libexec/git-core/git switch -C fork-1 fork/fork-1
21:25:01.088263 git.c:476               trace: built-in: git switch -C fork-1 fork/fork-1
21:25:01.090919 run-command.c:667       trace: run_command: 'git-lfs filter-process'
21:25:01.090928 run-command.c:759       trace: start_command: /bin/sh -c 'git-lfs filter-process' 'git-lfs filter-process'
21:25:01.125370 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
21:25:01.132484 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
21:25:01.137705 trace git-lfs: exec: git 'config' '--includes' '--local' 'lfs.repositoryformatversion'
21:25:01.143123 trace git-lfs: exec: git 'config' '--includes' '-l'
21:25:01.149573 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
21:25:01.155445 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
21:25:01.162004 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
21:25:01.167965 trace git-lfs: Install hook: pre-push, force=false, path=lfs-fork-test-ori/.git/hooks/pre-push, upgrading...
21:25:01.168031 trace git-lfs: Install hook: post-checkout, force=false, path=lfs-fork-test-ori/.git/hooks/post-checkout, upgrading...
21:25:01.168055 trace git-lfs: Install hook: post-commit, force=false, path=lfs-fork-test-ori/.git/hooks/post-commit, upgrading...
21:25:01.168076 trace git-lfs: Install hook: post-merge, force=false, path=lfs-fork-test-ori/.git/hooks/post-merge, upgrading...
21:25:01.168100 trace git-lfs: Initialize filter-process
21:25:01.168995 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
21:25:01.174505 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir'
21:25:01.181177 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
21:25:01.187266 trace git-lfs: tq: running as batched queue, batch size of 100
21:25:01.187368 trace git-lfs: filepathfilter: accepting "test.lfs"
21:25:01.187509 trace git-lfs: tq: sending batch of size 1
21:25:01.187647 trace git-lfs: api: batch 1 files
21:25:01.187814 trace git-lfs: HTTP: POST https://git.pernicious.games/try-gitea-lfs/lfs-fork-test.git/info/lfs/objects/batch
21:25:01.780130 trace git-lfs: HTTP: 200
21:25:01.783595 trace git-lfs: HTTP: {"objects":[{"oid":"7ffc8bb3b0c443a3d170b1f6aa402a66c6de86c49dd2d74e452d6d82b855939f","size":9,"error":{"code":404,"message":"Not Found"}}]}

21:25:01.783875 trace git-lfs: tq: starting transfer adapter "basic"
21:25:01.784456 trace git-lfs: tq: running as batched queue, batch size of 100
21:25:01.785535 trace git-lfs: filepathfilter: accepting "test.lfs"
Downloading test.lfs (9 B)
21:25:01.785616 trace git-lfs: tq: running as batched queue, batch size of 100
21:25:01.785902 trace git-lfs: tq: sending batch of size 1
21:25:01.786536 trace git-lfs: api: batch 1 files
21:25:01.786627 trace git-lfs: HTTP: POST https://git.pernicious.games/try-gitea-lfs/lfs-fork-test.git/info/lfs/objects/batch
21:25:01.861974 trace git-lfs: HTTP: 200
21:25:01.862152 trace git-lfs: HTTP: {"objects":[{"oid":"7ffc8bb3b0c443a3d170b1f6aa402a66c6de86c49dd2d74e452d6d82b855939f","size":9,"error":{"code":404,"message":"Not Found"}}]}

21:25:01.862324 trace git-lfs: tq: starting transfer adapter "basic"
Error downloading object: test.lfs (7ffc8bb): Smudge error: Error downloading test.lfs (7ffc8bb3b0c443a3d170b1f6aa402a66c6de86c49dd2d74e452d6d82b855939f): [7ffc8bb3b0c443a3d170b1f6aa402a66c6de86c49dd2d74e452d6d82b855939f] Not Found: [404] Not Found
21:25:01.865697 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir'
21:25:01.879004 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'

Errors logged to 'lfs-fork-test-ori/.git/lfs/logs/20250613T212501.862587.log'.
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: test.lfs: smudge filter lfs failed

Although I checkout a branch from the remote fork, but it will download the lfs object from the base repository https://git.pernicious.games/try-gitea-lfs/lfs-fork-test.git/info/lfs/objects/batch. Maybe this is a bug of git-lfs. It should get the url of the remote fork and download the lfs object from there. For those fork repositories' branches which have a PR in the base repository, the lfs pointers in that branch should be copied to base repository. But for other branches in the fork repositories, git-lfs should download lfs objects from the fork repository.

Member

OK. I figured it out. Once the git repository have multiple remotes, configure like below.

git config lfs.remote.autodetect true

Then the problem will be resolved.
ref: git-lfs/git-lfs#5988 (comment)
ref: https://github.com/git-lfs/git-lfs/blob/0534b10d870acd31b13ea2c23d94a710b44ab98f/docs/man/git-lfs-config.adoc?plain=1#L59-L64

Member

Since the table LFSMetaObject hasn't store the branch name. We cannot easily copy the pointers of that branch when creating the pull request.

type LFSMetaObject struct {
	ID           int64 `xorm:"pk autoincr"`
	lfs.Pointer  `xorm:"extends"`
	RepositoryID int64              `xorm:"UNIQUE(s) INDEX NOT NULL"`
	Existing     bool               `xorm:"-"`
	CreatedUnix  timeutil.TimeStamp `xorm:"created"`
	UpdatedUnix  timeutil.TimeStamp `xorm:"INDEX updated"`
}

Maybe two columns path and branch needs to be stored in this table or a new table.

type LFSMetaObject struct {
	ID           int64 `xorm:"pk autoincr"`
	lfs.Pointer  `xorm:"extends"`
        Branch string
        Path string
	RepositoryID int64              `xorm:"UNIQUE(s) INDEX NOT NULL"`
	Existing     bool               `xorm:"-"`
	CreatedUnix  timeutil.TimeStamp `xorm:"created"`
	UpdatedUnix  timeutil.TimeStamp `xorm:"INDEX updated"`
}

Contributor

This LFS association stuff is general issue. To me, if you know the pointer of LFS object, you should be able to download it. The pointer is already unique and if we have a collision, then there are bigger problems than Gitea.

Adding a branch here makes things much more complicated -- what happens if you have LFS objects on branch

repo.branch1

Then you merge them to

repo.branch2

And then maybe you cherry pick these commits to another branch and push them too. Do you then check every commits for all LFS objects and try to associate them with every branch? What happens if you just fetch a SHA object and don't care about a branch.

git fetch --depth=1 remote SHA

IFF Gitea is doing these operations, sure, it's then "easy" to copy stuff around. But git is decentralized and these workflows are mostly not done on a Gitea instance but on some client.

A better way would be to associate objects by inspecting LFS objects on every push... but... there's an authentication issue that remains. IFF you know the Hash+Size, should that be sufficient to fetch the source object??

In my thinking, there's a case for removing the RepositoryID association with LFS object altogether. IFF you know the Hash+Size, then you already have a Git checkout of the sources. So you have access to the sources. The LFS<->RepositoryID association is then mostly unnecessary overhead. To put it another way, consider the world without LFS. In that world, all objects are in Git. A tarball is then a Git blob you get during fetch. Done. Now, in an LFS world, the LFS object you fetch on checkout -- but if you know the HASH/size of the hashed object, it means you already have the Git sources -- so you were authenticated to fetch the sources (from somewhere). Therefore there should be no additional authentication needed to download the object as you already had access to fetch its "key".

LFS shouldn't be adding hurdles to fetch complete objects.

PS. I'm assuming that git-lfs oids are always only full sha256 oids, and not some weird short oids.

Member

One benefit of including the branch and path in the table is that it allows displaying an lfs label in the file list, similar to how GitLab and Hugging Face do it.

https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/197

Contributor

It would be "easier" to list files with the tag, yes. But then we'd have this information in two places -- the gitea database and the Git store. That's a recipe for getting out of sync.

It may be better to parse the .gitattributes and then add the [lfs] tag on the fly for files that are correct format. We could even warn, like LFS does on checkout, if files are suppose to be tracked according to .gitattributes but are not in LFS. For this, all that needs to be done is checking the first few bytes of the file. For example,

version https://git-lfs.github.com/spec/v1
oid sha256:fd298d3599e673384ac1f903cbbb436b8b1f4375eacc78a59f42b3502688d3f1
size 84

And that is just the first 128 bytes. So, reading first up-to-150 bytes or so of the blob is enough to read the entire LFS pointer and verify that object is suppose to be in LFS or not. Nothing external to just git cat-file required.

Aside: For GitLab, someone reported similar behaviour I proposed earier (no repository association) as a "vulnerability" (and other issues)

I will discuss this with our security guys but I'm heavily leaning that this is not a vulnerability as otherwise all of TLS is basically broken too with same argument. The main reason why we want authentication for LFS uploads/fetches is to prevent LFS server to be abused as a type of remote file store for non-git repositories and to be able to track uploads.

Member

The git check-attr command is not efficient, especially when trying to identify LFS-tracked files within a specific subdirectory. It requires checking each file individually, which can be slow and cumbersome for large trees, as we must invoke git check-attr on every file to determine whether it matches the LFS attributes.

ContributorAuthor

The fix is known, if you're okay with re-enabling the automatic LFS file linkage. I've been maintaining it in my fork for a while: parnic-sks@b718e44

which is essentially a revert of 2bb3200 , if I'm remembering correctly.

Member

I think the patch might work, but it could also introduce inconsistent or “dirty” LFS metadata that doesn’t belong to the target repository. Ideally, this should be handled by copying the relevant LFS metadata from the fork’s branch to the base repository’s branch at the time the pull request is created—similar to how Git handles regular data.

However, since the lfs_meta_object table doesn’t include a branch reference, it’s difficult to determine which LFS pointers should be copied. One possible workaround is to copy all LFS pointers from the fork repository when the pull request is created.

As for cleanup, we also need to consider how to safely remove these LFS pointers when the fork repository is deleted—especially those that are not referenced in the base repository.