x/tools/gopls: On ANY network error, error says GOPROXY=off

[WhyNotHugo]

What version of Go are you using (go version)?

go version go1.18.3 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/hugo/.cache/go-build"
GOENV="/home/hugo/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/hugo/.cache/golang/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/hugo/.cache/golang"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18.3"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1562160563=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I tried to use the LSP. By mistake, the process had no network access.

What did you expect to see?

The LSP should indicate that there is a network error.

What did you see instead?

The LSP indicated that GOPROXY=off, which made me assume the binaries are was using where messed up or reading GOPROXY from some obscure place:

gopls: 0: err: exit status 1: stderr: go: github.com/aws/[email protected]: module lookup disabled by GOPROXY=off          
go: github.com/aws/[email protected]: module lookup disabled by GOPROXY=off

The logs also indicate the same (though have a single mention of network error between a dozen of the following):

[ERROR][2022-07-04 10:33:35] ...lsp/handlers.lua:454	"2022/07/04 08:33:35 go/packages.Load: err: exit status 1: stderr: go: github.com/aws/[email protected]: module lookup disabled by GOPROXY=off\ngo: github.com/aws/[email protected]: module lookup disabled by GOPROXY=off\n\n\tsnapshot=1\n\tdirectory=/home/hugo/clones/gitlab.com/proctorexam1/tools/tailor\n\tquery=[file=/home/hugo/clones/gitlab.com/proctorexam1/tools/tailor/main.go]\n\tpackages=0\n"

In reality, GOPROXY was unset; the error message is completely wrong here.

See: NixOS/nixpkgs#177632

Reproducing tips

You can repro this by running gopls in a network namespace with not real network access:

bwrap --ro-bind /path/to/project /path/to/project --ro-bind /usr /usr --ro-bind /lib64 /lib64 --proc /proc --dev /dev --tmpfs /tmp --new-session --die-with-parent -- gopls

You can also repro this by allowing gopls to have network access, but without access to /etc/resolv.conf:

bwrap --share-net --ro-bind /path/to/project /path/to/project --ro-bind /usr /usr --ro-bind /lib64 /lib64 --proc /proc --dev /dev --tmpfs /tmp --new-session --die-with-parent -- gopls

Also yields:

gopls: 0: err: exit status 1: stderr: go: github.com/aws/[email protected]: module lookup disabled by GOPROXY=off          
go: github.com/aws/[email protected]: module lookup disabled by GOPROXY=off

Finally, even when gopls has network access and can read /etc/resolv.conf, but has not access to /etc/ssl/certs/ca-certificates.crt, it will also show the same error:

gopls: 0: err: exit status 1: stderr: go: github.com/aws/[email protected]: module lookup disabled by GOPROXY=off          
go: github.com/aws/[email protected]: module lookup disabled by GOPROXY=off

[WhyNotHugo]

If GOPATH is read-only, gopls shows no error and the logs merely say:

[ERROR][2022-07-04 10:52:48] ...lsp/handlers.lua:454	"2022/07/04 08:52:48 tidy: diagnosing file:///home/hugo/clones/gitlab.com/proctorexam1/tools/tailor/go.mod: err: exit status 1: stderr: go: downloading github.com/go-errors/errors v1.0.1\ngo: downloading github.com/google/go-cmp v0.5.7\ngo: downloading github.com/pingcap/errors v0.11.4\ngo: downloading github.com/pkg/errors v0.9.1\ngo: downloading github.com/jmespath/go-jmespath/internal/testify v1.5.1\ngo: downloading golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd\ntailor imports\n\tgithub.com/aws/aws-sdk-go-v2/config tested by\n\tgithub.com/aws/aws-sdk-go-v2/config.test imports\n\tgithub.com/google/go-cmp/cmp: module lookup disabled by GOPROXY=off\ntailor imports\n\tgithub.com/getsentry/sentry-go tested by\n\tgithub.com/getsentry/sentry-go.test imports\n\tgithub.com/go-errors/errors: module lookup disabled by GOPROXY=off\ntailor imports\n\tgithub.com/getsentry/sentry-go tested by\n\tgithub.com/getsentry/sentry-go.test imports\n\tgithub.com/google/go-cmp/cmp/cmpopts: module lookup disabled by GOPROXY=off\ntailor imports\n\tgithub.com/getsentry/sentry-go tested by\n\tgithub.com/getsentry/sentry-go.test imports\n\tgithub.com/pingcap/errors: module lookup disabled by GOPROXY=off\ntailor imports\n\tgithub.com/getsentry/sentry-go tested by\n\tgithub.com/getsentry/sentry-go.test imports\n\tgithub.com/pkg/errors: module lookup disabled by GOPROXY=off\ntailor imports\n\tgithub.com/aws/aws-sdk-go/service/s3/s3manager imports\n\tgithub.com/aws/aws-sdk-go/aws/awsutil imports\n\tgithub.com/jmespath/go-jmespath tested by\n\tgithub.com/jmespath/go-jmespath.test imports\n\tgithub.com/jmespath/go-jmespath/internal/testify/assert: module lookup disabled by GOPROXY=off\ntailor imports\n\tgithub.com/aws/aws-sdk-go/service/s3/s3manager imports\n\tgithub.com/aws/aws-sdk-go/service/s3 tested by\n\tgithub.com/aws/aws-sdk-go/service/s3.test imports\n\tgithub.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamtest imports\n\tgolang.org/x/net/http2: module lookup disabled by GOPROXY=off\n\n"

[hyangah]

If there was an underlying network issue, I think GOPROXY=off error is a red herring.

Except certain cases where module download is explicitly requested (e.g. go get or go mod tidy codelenses) or during initial package loading, gopls intentionally sets "GOPROXY=off" whenever it needs to use go list. That is intentional to avoid network access for every keystroke.

Users who want gopls to behave differently can try allowImplicitNetworkAccess - but still with network errors, I don't know exactly what error messages will popup.

I think there should be other errors during the initial workspace loading. Can you please capture the log from the gopls start and share the log?

@golang/tools-team we need to improve this error reporting - in particular, I think we should rewrite or hide error messages that include GOPROXY=off when gopls added that env var. Another possibly related issue is - gopls doesn't recover easily when initial workspace loading fails. #47540

[gopherbot]

Change https://go.dev/cl/419500 mentions this issue: internal/lsp/cache: allow network whenever reloading the workspace

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)