go build: get randomly stuck when building inside arm64 docker multi arch image

[slayerjain]

Go version

using the golang:1.22 docker image

Output of go env in your module/workspace:

hard to get since this is docker build phase.

What did you do?

1. I create a GitHub action that builds a multi arch docker image.
2. This GitHub actions runner is running on a self hosted linux machine. This machine is running a single node k8s cluster and using github actions controller to run multiple runners on this machine.
3. The docker image is being built for amd64 and arm64. the amd64 build never gets stuck but the arm64 gets stuck the first time and then builds fine after restarting the GitHub actions.
4. This the command: go build -tags=viper_bind_struct -ldflags="-X main.dsn=*** -X main.version=0.8.1 -X main.api_server_uri=https://api.***.io/" -o ***-enterprise .

What did you see happen?

The go build commands is stuck for 1 hour+ . The entire GHA build completes in less than 6 mins usually.

PFA the log file
go-build-logs.txt

What did you expect to see?

The build should be completed in less than 60 seconds - which does happen 50% of the time.

[gabyhelp]

Related Issues and Documentation

• go build hangs forever in 1.21.x #64705 (closed)
• cmd/go: "go build" indefinitely stuck due to invoking git #59103 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

how is it doing a cross compilation, is this using go's native support, or running on qemu, or something else?

where exactly is it getting stuck (use the -x flag)?

[slayerjain]

It's using docker buildx which seems to be running qemu to create an arm64 build container. I will add -x flag and report back

[seankhliao]

see also #67355

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

[weberc2-tempus]

We are seeing this as well when building with buildkit. When we send SIGQUIT to the child process, we get this stack trace
output.txt