cmd/dist: tools built with GOROOT_FINAL cannot locate it when copied to another location

[g4s8]

What version of Go are you using (go version)?

$ go version
go version go1.21.0 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ /usr/lib/go/bin/go env
GO111MODULE=''

GOARCH='amd64'

GOBIN=''

GOCACHE='/home/g4s8/.cache/go-build'

GOENV='/home/g4s8/.config/go/env'

GOEXE=''

GOEXPERIMENT=''

GOFLAGS=''

GOHOSTARCH='amd64'

GOHOSTOS='linux'

GOINSECURE=''

GOMODCACHE='/home/g4s8/go/pkg/mod'

GOOS='linux'

GOPATH='/home/g4s8/go'

GOPROXY='https://proxy.golang.org,direct'

GOROOT='/usr/lib/go'

GOSUMDB='sum.golang.org'

GOTMPDIR=''

GOTOOLCHAIN='auto'

GOTOOLDIR='/usr/lib/go/pkg/tool/linux_amd64'

GOVCS=''

GOVERSION='go1.21.0'

GCCGO='gccgo'

GOAMD64='v1'

AR='ar'

CC='gcc'

CXX='g++'

CGO_ENABLED='1'

GOMOD='/dev/null'

GOWORK=''

CGO_CFLAGS='-O2 -g'

CGO_CPPFLAGS=''

CGO_CXXFLAGS='-O2 -g'

CGO_FFLAGS='-O2 -g'

CGO_LDFLAGS='-O2 -g'

PKG_CONFIG='pkg-config'

GOGCCFLAGS='-fPIC -m64 -pthread -fmessage-length=0 -ffile-prefix-map=/tmp/go-build4273986275=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I'm trying to build go 1.21.0 from sources for Void Linux go package: https://github.com/void-linux/void-packages

The steps to build it (inside go repo directory):

unset GCC CC CXX LD CFLAGS
unset CGO_CXXFLAGS CGO_ENABLED

export GOROOT_FINAL=/usr/lib/go
export GOROOT=$PWD
export GOARCH=amd64

cd src
bash make.bash -v

sudo rm -f /usr/bin/go
sudo rm -fr /usr/lib/go/
sudo cp  ../bin/* -t /usr/bin
sudo cp -r .. /usr/lib/go

What did you expect to see?

When running /usr/bin/go version (or any other go command, like go build, etc) to see

go version go1.21.0 linux/amd64

What did you see instead?

When running go command or /usr/bin/go:

$ go version
go: cannot find GOROOT directory: 'go' binary is trimmed and GOROOT is not set

$ go build
go: cannot find GOROOT directory: 'go' binary is trimmed and GOROOT is not set

If I run /usr/lib/go/bin/go, then all commands works fine:

$ /usr/lib/go/bin/go version
go version go1.21.0 linux/amd64

If I run exactly the same build commands for go version 1.20.7, it's working fine:

$ go version
go version go1.20.7 linux/amd64

or

$ /usr/bin/go version
go version go1.20.7 linux/amd64

[mknyszek]

CC @bcmills @matloob via https://dev.golang.org/owners

[bcmills]

This is almost certainly a consequence of https://go.dev/cl/463740 (for #24904).

When built with -trimpath, the go command uses os.Executable to find its GOROOT, expecting that it is installed at $GOROOT/bin/go, and lots of Go programs (especially tests) expect to be able to find the go command at that location (from a configuration that gives only a GOROOT path).

@g4s8, is there a reason your script is using cp to make copies of the binaries in GOROOT/bin, instead of using ln to make /usr/bin/go a symbolic link? It seems like that would be a simpler solution, and would allow the Go toolchain installed on Void Linux to use the same reproducible binaries that are distributed through the Go project.

[bcmills]

I wonder if we should change cmd/dist to omit the -trimpath flag when GOROOT_FINAL is set.

(But I would still recommend not setting it, and using symlinks instead. 🤷‍♂️)

[bcmills]

I wonder if we should change cmd/dist to omit the -trimpath flag when GOROOT_FINAL is set.

That's extra tricky because I think the releasebot sets GOROOT_FINAL explicitly. 🤔

@golang/release, how much work would it be to stop setting GOROOT_FINAL in the environment for releases at Go 1.21 and later?

[dmitshur]

It wouldn't be much work at all. (The relevant code is in BuildletStep.makeEnv.) If it no longer needs to be set and dropping it is safe, I think we should get rid of it (and I'd be happy to help or do it).

Is #51504 relevant to this issue?

[bcmills]

Is #51504 relevant to this issue?

I don't think it's particularly closely related. The problem there occurs when running tests before moving the binaries to GOROOT_FINAL, but the problem here is running go commands after copying the binaries to their final locations (and copying the go executable to some other location outside of GOROOT_FINAL).

[rsc]

If /usr/bin/go is a symlink to /usr/lib/go/bin/go, I believe that will work correctly. Can your packaging do that?

It would be ideal if we could remove GOROOT_FINAL entirely. The distribution builds are position-independent and therefore reproducible. That's an important feature, and it seems like a mistake to have a flag to turn that off. However, we also want people to be able to package Go for Linux distributions. :-)

[gopherbot]

Change https://go.dev/cl/518837 mentions this issue: cmd/dist: omit -trimpath if GOROOT_FINAL is set

[g4s8]

Linking /usr/bin/go -> /usr/lib/go/bin/go works fine. Maybe this line should be removed (or updated?) from src/make.bash to avoid confusion?

# GOROOT_FINAL: The expected final Go root, baked into binaries.
# The default is the location of the Go tree during the build.

[bcmills]

Maybe this line should be removed (or updated?) from src/make.bash to avoid confusion?

I think we should remove it from pretty much everything, for which I have filed proposal #62047.

[kat-co]

If /usr/bin/go is a symlink to /usr/lib/go/bin/go, I believe that will work correctly. Can your packaging do that?

I have just done something very similar while packaging v1.21.0 for Guix, and it appears to have worked. The only remaining issue I can foresee is if other Guix maintainers have an issue with the new link structure.

[tspearconquest]

I'm building 1.21 from source on alpine using the 1.20 docker-library dockerfile customized to pull down the 1.21 sources rather than the binary package: https://github.com/docker-library/golang/blob/master/1.20/alpine3.18/Dockerfile but with line 13 changed to 1.21.0

Certain packages like golang.org/x/net are included in the base distribution and when there is a CVE identified, I wanted to be able to patch it just before running make.bash rather than making the dev team have to think about/manage CVEs in the base image they use to build their apps, which is why I'm avoiding the binary packages and building from source. This is for an internal build; not anything published publicly.

When I build 1.21.0, I'm seeing this issue. The dockerfile doesn't appear to set GOROOT or GOROOT_FINAL. It simply sets PATH=/usr/local/go/bin:$PATH early; way before the build happens, and then installs go into /usr/local/go.

Is the right fix for me to just add ENV GOROOT=/usr/local/go near the end of the dockerfile, perhaps at line 112?

[bcmills]

@tspearconquest, as far as I can tell that Dockerfile is not moving the final binaries, and should be resolving go to /usr/local/go/bin/go, from which it should be able to find GOROOT using os.Executable().

That is: the problem you are reporting is not obviously related to this one.

Could you please file a new issue with more detail on the failure mode you're seeing?

[tspearconquest]

Sure; I did simplify the customizations a bit. We use a multi-stage docker build; so we're doing COPY /usr/local/go /usr/local/go in the final stage, and as a guess, since make.bash is setting GOROOT_FINAL itself but GOROOT is not being re-set in the final stage, that's probably why go is giving this error. Full context in #62119

[bcmills]

#62047 is approved, so the resolution here is that we will remove support for GOROOT_FINAL rather than making it work for copied binaries.