Description
What version of Go are you using (go version)?
1.15.6 before that we used 1.12.*
Does this issue reproduce with the latest release?
1.15.6 is latest, so yes.
What operating system and processor architecture are you using (go env)?
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS="-v -trimpath"
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/builds/symflower/symflower/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/builds/symflower/symflower"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang-11"
CXX="clang++-11"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build695024912=/tmp/go-build -gno-record-gcc-switches"
What did you do?
(This issue is created because of #40461 where i asked if should create an issue or if it fits the referenced one.)
Gist: Parallel running "go build" processes that use the same GOCACHE directory lead to errors such as crypto/tls\n/usr/local/go/src/crypto/tls/common.go:22:2: can't open import: \"net\": open /cache/go-build/87/8713176dd630051cd4b09a7fa534da9a291da82ba5c84c02966f3ee18dc3a206-d: no such file or directory.
Longer: We switched from Go 1.12 to 1.15.6 for one project and are now running into constant non-deterministic build/test problems because of what I think is that Go's cache is not concurrent-safe. Basically our CI does one "go build" over a whole branch and uses one common GOCACHE per CI node. Hence, multiple CI jobs could write to the same cache directory. That part was never a problem even with the newer Go version (i do not know why). After the "go build" CI job multiple jobs run in parallel to do linting and testing (via "go test"). And these run into constant issues e.g. crypto/tls\n/usr/local/go/src/crypto/tls/common.go:22:2: can't open import: \"net\": open /cache/go-build/87/8713176dd630051cd4b09a7fa534da9a291da82ba5c84c02966f3ee18dc3a206-d: no such file or directory.
Some additional information that might be interesting:
- The CI runs in Docker containers over Kubernetes
- The GOCACHE points to a directory on an EXT4 partition which is local and not synced over the network.
What did you expect to see?
We expected that all builds can use the same cache directory at the same time and the builds that worked with an older Go version still work with the latest version.
What did you see instead?
Hundreds of errors such as crypto/tls\n/usr/local/go/src/crypto/tls/common.go:22:2: can't open import: \"net\": open /cache/go-build/87/8713176dd630051cd4b09a7fa534da9a291da82ba5c84c02966f3ee18dc3a206-d: no such file or directory. That lead to a failing build. We never had this problem with Go 1.12. The cache just worked. With 1.15.6 it does not. Basically because of this problem we are back to having no cache at all.
We do not run "go clean" at all, so this is hopefully not #31948.
Our current workaround:
- Use a GOCACHE directory inside of the running container. The "build" job copies the whole GOCACHE to all other CI jobs that then can reuse the cache. So the cache is only valid for one CI pipeline. This creates HUGE artifacts but is the only way we currently see to have a half-working Go build cache.