cmd/go: "go build" indefinitely stuck due to invoking git

[NuLL3rr0r]

What version of Go are you using (go version)?

$ go version
go version go1.20.2 linux/amd64

Does this issue reproduce with the latest release?

Yes!

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/mamadou/.cache/go-build"
GOENV="/home/mamadou/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/mamadou/dev/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/mamadou/dev/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.2"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="x86_64-pc-linux-gnu-gcc"
CXX="x86_64-pc-linux-gnu-g++"
CGO_ENABLED="1"
GOMOD="/home/mamadou/dev/MamadouArchives/MamadouArchives/Source/ThirdParty/MamadouArchivesBackend/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build280206031=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Well, I have a heavy Git/LFS repository hosting Unreal Engine + a Game project that relies on some utilities written in Golang. It was fine until I created another branch with the latest version of the engine and merged that to the master branch. Now, the issue is not Unreal Engine or Git, but go build. When I invoke go build command from my Makefile it gets stuck forever. Then I tried to track down the issue, so in that go module I invoke:

$ go build -x
WORK=/tmp/go-build3881985498
cd /home/mamadou/dev/MamadouArchives
git status --porcelain
cd /home/mamadou/dev/MamadouArchives
git -c log.showsignature=false show -s --format=%H:%ct

It's been stuck for 10 minutes now and never finished. Then, I onlly copy the Go module anywhere outside the Git repo and go build -x produces the binary just fine.

What did you expect to see?

I guess it should not be stuck. A timeout? Maybe! Though, I will try to leave it over night to see if it ever finishes.

What did you see instead?

Nothing! It just hangs forever.

[cherrymui]

cc @bcmills @matloob

[cherrymui]

@NuLL3rr0r does this reproduce with Go 1.19 or other version of Go? Thanks.

[NuLL3rr0r]

I don't have any other versions installed. Though, I left the PC for around half an hour and when I came back it was done as expected. Probably Git has done some background work. Now, when I invoke make it just builds as usual. But, I guess this is still a valid issue for huge repositories like mine, and I'd like to be able to at least provide a timeout for such a operations, e.g. through an env var. Though, feel free to close it if you don't think so.

[seankhliao]

since you're saying git-lfs, I guess it's downloading things into the module cache
so subsequent runs are faster.

you could use go build -x with a clean (or temporary) module cache to see what's sloe

[NuLL3rr0r]

The repo did not download anything. I am using i3wm and always have my dl/up speed at the top bar and I actually had an eye on that to see if it's downloading. Not to mention I am the sole user of this repo and there was no new commits.

As I said, I am suspecting the culprit might be Git housekeeping. Also, my filesystem is ZFS if it makes a difference that might as well do some housekeeping.

When originally, I merged the new branch it took around 14/15 hours to merge the updated branch to master. Unreal Engine is huge just checkout their repo. I guess I had just one commit with over 160K files modified.

My point is it's good to have a timeout. I might be wrong and I am the only one who keep their golang tools in the same repo as the whole engine and the game. But, this is common practice at a few game studios I have worked for as well. Though the ususally use SVN or Perforce.

[bcmills]

If the operation will eventually succeed, then I would rather not introduce a timeout that would cause it to spuriously fail; you can run go build with -buildvcs=false to avoid stamping Git metadata in the first place.

FWIW, there is a Linux timeout command that you can use to send a signal if the command is taking too long.

The go command should respond appropriately to SIGINT and SIGTERM. (If it doesn't today that's a clear bug we should fix.) So you should be able to use timeout --signal=INT 10m go build … to get the behavior you want. Please give that a try and let us know how it goes.

[NuLL3rr0r]

Thank you very much for the extra information. The only way for me to reproduce this is basically to clone GBs of data and revert back to older versions of the project. Or, I could wait until the next major Unreal upgrade is stabilized then I definitely have to go through the same procedure. So, it might take a while before I could provide that info.