cmd/go: suppress errors for 'go get' of package paths whose source files are all excluded by build constraints

[jeanbza]

What version of Go are you using (go version)?

$ go version
go version go1.13beta1 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/deklerk/Library/Caches/go-build"
GOENV="/Users/deklerk/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/deklerk/workspace/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/tmp/foo/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/c2/cvxltzcd66v5lx14hm1j76q000h16k/T/go-build777475143=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

deklerk at deklerk-macbookpro2 in /tmp/foo
$ go get google.golang.org/[email protected]
go: downloading golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961
go: downloading golang.org/x/tools v0.0.0-20190226205152-f727befe758c
go: downloading honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099
go: extracting honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099
go: extracting golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961
go: extracting golang.org/x/tools v0.0.0-20190226205152-f727befe758c
can't load package: package google.golang.org/genproto: build constraints exclude all Go files in /Users/deklerk/workspace/go/pkg/mod/google.golang.org/[email protected]
deklerk at deklerk-macbookpro2 in /tmp/foo

What did you expect to see?

deklerk at deklerk-macbookpro2 in /tmp/foo
$ go get google.golang.org/genproto/...
go: downloading golang.org/x/sys v0.0.0-20180830151530-49385e6e1522
go: extracting golang.org/x/sys v0.0.0-20180830151530-49385e6e1522
go: finding google.golang.org/genproto latest
deklerk at deklerk-macbookpro2 in /tmp/foo

What did you see instead?

(see above)

TLDR genproto has a build constraint at the root (+tools, nothing else at root), but many packages below the root do not have this build constraint. When I use go get in module mode - esp when I provide a version - I expect to only interact with the module system. Instead, it's a weird mix of module and package, and in this case though I wanted to get a module it gave me an error about a package of the same name.

[bcmills]

go get requests to build a binary. If you know that what you want is just the module (and not the binary at the root of that module), go get -d informs the go command that it should stop without attempting to build such a binary.

It may be that we should suppress the error without -d anyway, though. (See #29268, which is very closely related.)

[jayconrod]

#40145 is a duplicate of this issue, but that case seems more serious. go get -d fails when given a module path where the module root directory contains ignored .go files. The intent of the command is to update the dependency, not to build anyway.

I think this is happening because go get tries to decide fairly early whether each argument refers to a package or a module. If an argument is a module path, and there are no .go files at the module root, it's assumed to be a module path, and go get will not attempt to load a package. In this case, there are .go files, even though they're ignored, so go get tries and fails to load the package.

go get should probably handle the error from modload.ImportPathsQuiet more gracefully in this case.

[mvdan]

Thanks, @jayconrod! Should I ping the protobuf maintainers about this? If this won't be fixed until 1.16, they probably want to work around the issue by moving the file until most developers are on 1.16 or later. FYI @dsnet

[jayconrod]

I think it would be good to put in a workaround for this. The file in question, integration_test.go seems like it could be moved out of the root directory.

Unfortunately, I think we're too deep in the freeze to land a bug fix for 1.15. Hopefully though that means 1.15 will be out on time and we'll have a longer development window for 1.16!

[bcmills]

As of Go 1.16, go get now does suppress this error if the -d flag is set (see CL 289769), but not otherwise (because go get is being asked to actually build the package).

We should probably suppress it when the -d flag is not set, too, since we're phasing out the need for -d in general (see #43684).

[gopherbot]

Change https://golang.org/cl/297009 mentions this issue: cmd/go: suppress errors for 'go get' of module paths that are also constrained-out packages

[mvdan]

@bcmills @jayconrod I'm encountering this issue again with go get -d on Go 1.16.4, so I'm a bit confused:

$ go version
go version go1.16.4 linux/amd64
$ go get -d mvdan.cc/sh/[email protected] # no "+build gofuzz" file
go: downloading mvdan.cc/sh/v3 v3.2.0
$ go get -d mvdan.cc/sh/[email protected] # "+build gofuzz" file appears here
mvdan.cc/sh/v3: no Go source files

Edit: I should note that go version devel go1.17-ce92a2023c Sat May 15 02:39:08 2021 +0000 linux/amd64 works fine there.

[bcmills]

@mvdan, interesting! I see the same behavior with go1.16.4 but not with gotip:

$ go version
go version devel go1.17-8b0901fd3 Tue May 18 07:50:25 2021 +0000 linux/amd64

$ go get -d mvdan.cc/sh/[email protected]
go: downloading mvdan.cc/sh/v3 v3.3.0
go get: added mvdan.cc/sh/v3 v3.3.0

$ go1.16.4 get -d mvdan.cc/sh/[email protected]
mvdan.cc/sh/v3: no Go source files

$

So we must have fixed something else during the 1.17 cycle, but I'm not sure what.

[mvdan]

Should we reopen this temporarily while we investigate, or open a new issue about it? I'll tend to forget about these things if they're buried in a closed issue :)

[bcmills]

Since this doesn't reproduce at head, I'd say no: I don't think we would backport a fix even if we found it, given that it's fairly easy to work around. (You can go get -d the specific package(s) you need instead of the module root.)

[bcmills]

(If it were a regression from 1.15 to 1.16 that would be another matter, but the situation was even worse on 1.15... 😅)

[mvdan]

Fair enough, I'll add a workaround on my end for now - since I do think it's reasonable to go get -d module-path, and I do it pretty often :)