x/tools/go/analysis/analysistest: support modules

[muirdm]

Currently the analysistest helper only supports a GOPATH like tree. This is fine for stand alone analyses, but does not help much for analyses designed to run against a particular module. For example, you have an analysis that checks all code in your module correctly/safely interacts with a specific package in your module. Ideally you would be able to write an analysis test with a stub file that is able to load other packages/dependencies of the current module.

/cc @matloob @heschik

[matloob]

Just to understand this a bit better, do you mean that the test code in the test tree can depend on an external module and download it?

[muirdm]

do you mean that the test code in the test tree can depend on an external module and download it

I mean the test code behaves as if it is part of the module that contains the analyzer. I'm still getting used to modules, but I envisioned it working one of two ways:

1. I don't use analysistest.WriteFiles, instead writing my own temporary test package inside my module. Then I run analysistest.Run on my test package, and any imports in my test package resolve in module mode. If I understand, this doesn't work because analysistest puts go/packages into GOPATH mode.
2. I do use analysistest.WriteFiles and it creates a temporary module with a replace directive pointing back to my containing module.

[matloob]

Hm, I think Option 1 would create some problems because running the test would modify the module is contained in, and that could be very surprising.

Option 2 would be more desirable depending on how it works.

[bflad]

Hi there! 👋 I'm not sure if this is an intended use case, but we love that we can write analysis.Analyzer for other Go modules using real dependencies in the testing code rather than stubbing them, which can be quick complex. This requires some careful dependency handling with vendoring and symlinks to make it work with analysistest.Run. Since files are copied between test runs its a little slow though. Full implementation: https://github.com/bflad/tfproviderlint / Example Analyzer: https://github.com/bflad/tfproviderlint/tree/master/passes/S037

It'd be neat if this could use something like Go 1.14's -modfile support or it just allowed using a go.mod in the testdata source. In our case, we would just point all our testing at the same Go Module file (contents at least).

[matloob]

I'd welcome a contribution to fix this if the code is simple enough. My preference is that there's no extra configuration other than the module file of the module the tests are contained in.

[firelizzard18]

I am writing an analyzer that verifies that the correct packages are imported. I have files in testdata/ that import github.com/go-gl/gl/<version>/gl. go mod tidy does not add a reference to the module, github.com/go-gl/gl, and in fact it will remove a manually added reference. But when I run tests via analysistest.Run(t, analysistest.TestData(), pkgvet.Analyzer), I get the following error:

import_gl2.1.go:3:8: cannot find package "github.com/go-gl/gl/v4.6-core/gl" in any of:
        /usr/lib/go/src/github.com/go-gl/gl/v4.6-core/gl (from $GOROOT)
        REDACTED/src/go-pkgvet/testdata/src/github.com/go-gl/gl/v4.6-core/gl (from $GOPATH)

So in my case "analysistest: support modules" means it knowing how to look in the module cache for package sources.