x/tools/cmd/godoc: add module support

[dmitshur]

This is the tracking issue specifically for the golang.org/x/tools/cmd/godoc command to have support for displaying package documentation in module mode. The parent umbrella issue for all “godoc”-like projects is #26827.

At this time, the godoc command is only able to display packages that are located in a GOPATH workspace. When invoked in module mode, it should be able to show documentation for packages in the build list of the main module (as can be determined by running go list -m all), even if those packages are not inside a GOPATH workspace.

The plan is to make this change while maintaining GOPATH support. godoc will use the same logic as the go command to determine whether to use module mode or GOPATH mode (i.e., the value of GO111MODULE environment variable and the presence of a go.mod file).

Scope

There may be future enhancements to add features that weren’t previously viable in GOPATH mode, like being able to select the version via a dropdown UI element, but that’s not in scope for this issue.

This issue is only about maintaining the ability to view the documentation for packages when in module mode.

The -analysis flag is not in scope of this issue, since it's not required to view documentation. It will take more work and is being tracked separately in issue #34473.

[dmitshur]

Possible Solutions

I have spent some time investigating ways this issue can be resolved. There are many potential long-term solutions that involve making large changes, such as moving towards a newer documentation HTML renderer, making API changes to the godoc library and having it use go/packages to load packages, etc. There is also another solution that requires making a very small, targeted change that reuses much of existing godoc code and functionality. (Another upside is that it allows maintaining -analysis flag support with a small amount of additional effort, tracking issue #34473.) That is the solution I plan to use to resolve this issue.

Godoc functionality is currently spread between the golang.org/x/tools/cmd/godoc command, and the golang.org/x/tools/godoc library. The godoc library API operates on top of a virtual filesystem abstraction, which offers a healthy amount of flexibility. In fact, it's possible to add module support to the godoc command without making any changes to the godoc library API.

Implementation Plan

The goal is to make the godoc command run in module mode whenever the go command would run in module mode. To implement this reliably, godoc will run go env GOMOD with the same environment and working directory, and interpret the result. The -json flag will be used.

If the go env GOMOD result is the empty string, GOPATH mode is being used, and godoc will behave as it currently does: it will find each GOPATH workspace and bind it to the virtual filesystem:

// Bind $GOPATH trees into Go root.
for _, p := range filepath.SplitList(build.Default.GOPATH) {
	fs.Bind("/src", gatefs.New(vfs.OS(p), fsGate), "/src", vfs.BindAfter)
}

If the go env GOMOD result is a non-empty string, module mode is being used. godoc will use go list -m all to determine all modules in the build list, and bind their roots in appropriate locations within the filesystem:

// Determine modules in the build list.
type mod struct {
	Path    string // Module path.
	Version string // Module version.
	Dir     string // Directory holding files for this module, if any.
}
var mods []mod
[... populate mods with output from 'go list -m -json all']

// Bind module trees into Go root.
for _, m := range mods {
	if m.Dir == "" {
		[... either use 'go mod download' to fill module cache, or skip]
	}
	fs.Bind(path.Join("/src", m.Path), gatefs.New(vfs.OS(m.Dir), fsGate), "/", vfs.BindAfter))
}

This way, we outsource the work of doing version selection to the go command, which guarantees consistent results. It will result in a virtual filesystem that accurately contains the source code of Go packages that would be used in the build. It respects local replace directives in the go.mod file, if any. (It will not pick up changes to go.mod file that are made after godoc is started, that will take more changes.)

There may be small changes to the virtual filesystem implementation so that modules are not considered to be inside GOROOT. Some additional care may need to be taken to ensure overlapping paths, in case of nested modules, are supported.

There are tests in the golang.org/x/tools/cmd/godoc package that verify functionality. They will be converted to use packagestest and made to run in both GOPATH mode and module mode.

Decision on automatic downloading of modules not in module cache

Documentation for go list -m notes:

When listing modules, the -f flag still specifies a format template applied to a Go struct, but now a Module struct:

type Module struct {
    [...]
    Dir string // directory holding files for this module, if any
    [...]
}

An important part here is "if any". Most of the time, modules in the build list of the main module are available in the module cache, and the Dir field points to a location on disk where that module is available.

In some cases, for example when a new module has been downloaded but not yet built, some dependencies are not yet available in the module cache. The Dir field will be the empty string, and godoc will not be able to display documentation for that module.

A potential solution here is to simulate what the go command currently does in similar scenarios: automatically start to download the missing modules into the module cache. This can be implemented by invoking go mod download on modules in the build list that are not already present in the module cache.

The alternative solution is to rely on the user to manually run go mod tidy, go test ./..., go build ./..., or any other go command that would populate the missing module cache entries. I am still evaluating which solution to go with here, and currently leaning slightly towards the behavior that is consistent with the go command. Feedback on this decision is welcome.

/cc @andybons

[gopherbot]

Change https://golang.org/cl/196978 mentions this issue: godoc: remove Corpus.testDir field

[gopherbot]

Change https://golang.org/cl/196979 mentions this issue: cmd/godoc: check if server exited when waiting

[gopherbot]

Change https://golang.org/cl/196981 mentions this issue: cmd/godoc: convert tests to packagestest, cover third party packages