proposal: x/tools/go/analysis/testchecker: analysis driver for tests

[seankhliao]

Proposal Details

I propose a new helper for running x/tools/go/analysis/passes over the files in a module, and reporting the results as a test pass/fail.

// Test runs the given analyzers against the given package patterns.
// Each package matching the pattern and analyzer is reported as a sub test,
// in the format "<analyzer.Name>/<package path>".
// It fails the subtest if there are any findings.
func Test(t *testing.T, analyzers []*analysis.Analyzer, patterns []string)

Example usage:

import (
    "testing"

    "golang.org/x/tools/go/analysis"
    "golang.org/x/tools/go/analysis/passes/appends"
    "golang.org/x/tools/go/analysis/passes/asmdecl"
    "golang.org/x/tools/go/analysis/testchecker"
)

func TestLint(t *testint.T) {
    analyzers := []*analysis.Analyzer{
        appends.Analyzer,
        asmdecl.Analyzer,
    }
    testchecker.Test(t, analyzers, "./...")
}

Go has a rich ecosystem of static analysis, much of which is built on top of x/tools/go/analysis.
While these can aide developers while authoring code, like when integrated into gopls https://github.com/golang/tools/blob/master/gopls/doc/analyzers.md , many projects also enforce a common set of checks during CI.

Today, there are a variety of options to do that, for example, a random sampling:

• this repo
  • has a main package built using multichecker
  • builds and runs the analyzer in CI
  • (I think it's part of some online course, there are many copies of the same name / approach)
• this repo runs analysis binary in a test
• this repo installs and runs an analyzer directly (shadow)
• this repo runs a meta linter in git hooks and again as a ci jon
• this repo runs it in a makefile

Go already runs a set of vet checks by default during go test, these are high confidence checks that apply to everyone. Individual projects can usually be stricter and enforce more checks by default.
I think a helper to directly integrate linters into tests can ease the sprawl of complexity in ensuring projects continue to adhere to their desired quality gates.

[gabyhelp]

Related Issues

• x/tools/go/analysis/checker: an importable analysis driver based on go/packages #61324 (closed)
• proposal: x/tools/go/analysis/analysistest: improved, module-aware API #61336
• x/tools/go/analysis/analysistest: add module-aware analyzer test runners #53063
• x/tools/go/analysis: feature request: need a way of accessing analysis results from another go package #50265 (closed)
• proposal: x/tools/go/analysis: export internal/analysisflags and internal/checker #53336 (closed)
• x/tools/go/analysis: provide driver for running analysis programmatically #31897 (closed)
• proposal: x/tools/go/analysis: Expose something like internal/checker.Run #70152 (closed)
• x/tools/go/analysis/analysistest: Functionality for verifying SuggestedFixes and TextEdits #38044 (closed)
• x/tools/go/analysis/internal: Add an option to emit diagnostics for external packages #38177
• x/tools/go/{analysis,packages}: expose TypeErrors []types.Error fields #54619 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

I think I should fold this into #61336

[adonovan]

Interesting timing: we have been having a lot of conversations recently about the possibility of dynamically loaded analyzers that live in the module, enabling projects to define their own rules, either for internal housekeeping or to help their users detect misuse of their APIs. If done right, merely reading code that depends on package P would cause your editor to highlight misuses of P's API. Obviously there are a lot of technical problems to solve--compiling or interpreting; security; performance; UX; accessibility of the notation to non-experts--but a dynamic system would also be a solution to this problem.

[seankhliao]

I also saw a small ecosystem around linting terraform providers (which are usually a lot of crud code to turn declarative IaC to api calls) built around https://github.com/bflad/tfproviderlint
This seems to extend into the users part you mention.

Otherwise, most of what I've seen is around project internal housekeeping.