cmd/go: go mod download does not work with submodules

[marwan-at-work]

Summary

https://github.com/NYTimes/gizmo has two go.mod files: one at the root level and one inside the examples folder.

If I try to go mod download the top level module as such:

GO111MODULE=on go mod download -json github.com/NYTimes/gizmo/@v/v1.2.4

Everything works correctly, but the .zip file that gets created does not have the examples folder...I imagine this is expected behavior because the examples folder and all of its sub-packages have their own go.mod file independent of the root one.

Therefore, I would expect that go mod download github.com/NYTimes/gizmo/examples/@v/v1.2.4 to work properly but currently it's not working (see error below)

From the above behavior, I have two thoughts:

1. It would be nice if we can somehow conclude that a repository has multiple modules. This behavior comes in handy when it comes to code discovery and documentation websites such as when godoc.org supports modules. For example, godoc.org already has a Subdirectories section so it's become an expected behavior to see all subdirectories so we should at least see the examples folder or be hinted at a sub-module so we wouldn't be surprised if that folder is missing. I have a few ideas as to how we can find out about them, but first I'd like to see if that idea is well-accepted in the first place.

2. We shouldn't error out on go mod downloading a sub-module especially that it's not included in the root-level module zip file.

What version of Go are you using (go version)?

$ go version
go version go1.12 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
A8495:~ 208581$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/208581/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/208581/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/n4/35bgdnkd6vlgqrdzbyl0x1ycmhcndx/T/go-build121030453=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

go mod download -json github.com/NYTimes/gizmo/[email protected]

What did you expect to see?

The sub-module correctly downloaded to $GOPATH/pkg/mod

What did you see instead?

go: finding github.com/NYTimes/gizmo/examples v1.2.4
{
	"Path": "github.com/NYTimes/gizmo/examples",
	"Version": "v1.2.4",
	"Error": "unknown revision examples/v1.2.4"
}

[bcmills]

The v1.2.4 tag applies only to the module at the repo root. A module within the repo at the relative path examples/ would need a corresponding tag examples/v1.2.4.

[bcmills]

I think that makes this a duplicate of #30850, but please reopen if you disagree.

[marwan-at-work]

@bcmills that solves issue no.2 but not issue no.1 -- For issue 1: are there any conclusions on whether go mod downlaod (or any go sub-command) should be able to tell us if a module has sub-modules? Purely for discovery/documentation reasons.

Thanks :)

[bcmills]

The module proxy protocol doesn't include nested modules within the parent module, so in general there is no way for the go command to indicate that they exist without actively searching for them, and we don't want the behavior to differ between proxied and non-proxied go commands without a very compelling reason.

(We certainly do want to support module discovery and documentation, but go mod download is not a very good fit for that use-case. Stay tuned!)

[bcmills]

We certainly do want to support module discovery and documentation, but go mod download is not a very good fit for that use-case. Stay tuned!

https://index.golang.org/ now serves a feed of known module paths and versions at https://index.golang.org/index. That should be suitable for discovering modules programmatically — please file issues if you find that it is not.

[marwan-at-work]

@bcmills I imagine for say github.com/user/pkg we can ping index.golang.org/index to see if any of those lines include a github.com/user/pkg/ prefix, however there are two drawbacks:

1. If the module has not been discovered by index.golang.org yet, we're not gonna find submodules.
2. If the module is private and not public, there's no way to use index.golang.org.