x/tools/gopls: imports extremely slow with large module cache

[myitcv]

What version of Go are you using (go version)?

$ go version
go version devel +44c9354c5a Fri Jun 21 05:21:30 2019 +0000 linux/amd64
$ go list -m golang.org/x/tools
golang.org/x/tools v0.0.0-20190620191750-1fa568393b23
$ go list -m golang.org/x/tools/gopls
golang.org/x/tools/gopls v0.0.0-20190620191750-1fa568393b23

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN="/home/myitcv/gostuff/src/github.com/myitcv/govim/cmd/govim/.bin"
GOCACHE="/home/myitcv/.cache/go-build"
GOENV="/home/myitcv/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/myitcv/gostuff"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/myitcv/gos"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/myitcv/gos/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/myitcv/gostuff/src/github.com/myitcv/govim/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build977371100=/tmp/go-build -gno-record-gcc-switches"

What did you do?

The issue I keep tripping across is making refactoring changes that, for example, rename variables. If I fail to rename all instances of the variable (perhaps I've copy-pasted some code from somewhere), there's a chance I've left behind a selector expression with the old variable name... which is now treated by imports as if it were a qualified identifier for a package that has not yet been imported.

Like #32726, imports then heads off into the module cache searching through all potential matches. With a 5GB module cache, this is not a quick operation.

I think imports should:

• pre-load and cache its results from the module cache
• have that cache be invalidated where the modification time of any non-module cache-based go.{mod,sum} reachable from the main module changes
• when the cache is invalidated, re-warm the cache in the background

Yes, this will cause possibly stale results in the case that another main module causes changes in the module cache, but I think that's a sacrifice worth making for the general speed of imports.

What did you expect to see?

Instant responses to imports-based format-on-save.

What did you see instead?

~10sec delays.

---

cc @stamblerre @ianthehat @heschik

[bmhatfield]

Is there a command I can run on my machine to demonstrate if I am experiencing this? I only have a 250MB module cache, but imports take 30-45s on my machine at least. I do have a likely somewhat larger than normal repo with many packages in it, however.

[stamblerre]

Do you see this kind of latency when you run goimports via the command line or is it only when gopls invokes it?

[heschi]

@bmhatfield Run goimports -v on the file and see how long it's taking to do directory scans. It should be pretty clear from the output but if not attach it here and I'll take a look.

[bmhatfield]

I took a file that was having difficulty performing imports (30-45s in vscode "running save participants") and never actually completing the imports. I edited it into a state where VSCode reliably is sluggish and fails to perform imports/formatting (by deleting 3 import lines).

goimports -v runs in less than a second and the output contains the correct imports.

[stamblerre]

Do you see similar latencies with other gopls features, like jump to definition or autocomplete?

[bmhatfield]

Jump to definition: 2-3s latencies, with a blue "processing bar" that moves across the top of the window.
Autocomplete: feels instant.

[stamblerre]

Wow, that's a very strange range of latencies. Definition should definitely feel instant as well. I would have expected every feature to be slow rather than some, and then even so, it seems like go-to-definition is still faster than imports. Do you mind sharing a full gopls log (View: Debug Console -> Output -> Tasks -> gopls) just so I can get a fuller picture?

[bmhatfield]

Happy to - should we migrate this conversation to #32360, given that I am not experiencing the "large module cache" issue? Or prefer here still?