x/tools/gopls: quickfix_new_files (govim) test flaky

[myitcv]

What version of Go are you using (go version)?

$ go version
go version devel +8e0be05ec7 Sun Jan 19 14:04:09 2020 +0000 linux/amd64
$ go list -m golang.org/x/tools
golang.org/x/tools v0.0.0-20200117220505-0cba7a3a9ee9 => github.com/myitcvforks/tools v0.0.0-20200119092928-0fd5434cd1ba
$ go list -m golang.org/x/tools/gopls
golang.org/x/tools/gopls v0.1.8-0.20200117220505-0cba7a3a9ee9 => github.com/myitcvforks/tools/gopls v0.0.0-20200119092928-0fd5434cd1ba

Note that per govim/govim@c7a6cc3, github.com/myitcvforks/tools/gopls@v0.0.0-20200119092928-0fd5434cd1ba is 0cba7a3 with a partial revert of CL214586 on top (the result of which can be seen in https://go-review.googlesource.com/c/tools/+/215239)

The issue described below also exists when tested against 0cba7a3.

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/myitcv/.cache/go-build"
GOENV="/home/myitcv/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/myitcv/gostuff"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/myitcv/gos"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/myitcv/gos/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/myitcv/gostuff/src/github.com/myitcv/govim/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build026758154=/tmp/go-build -gno-record-gcc-switches"

What did you do?

We have just added a new govim test that seeks to verify diagnostics are as expected for new files that do not, initially, exist on disk.

The scenario involves creating a package p with a simple function DoIt. p is imported by a main package. p also has a test file that exercises DoIt, and an external test file that does the same.

Initially all call sites for DoIt incorrectly pass an integer argument, meaning we should have error diagnostics for all call sites. Then we correct the definition of DoIt to take an integer argument at which point all diagnostics should disappear.

The test can be seen here:

https://github.com/govim/govim/blob/cmd_govim_testscript_new_files_quickfix/cmd/govim/testdata/scenario_default/quickfix_new_files.txt

What did you expect to see?

1. Error diagnostics for main.go, p/p_test.go, and p/x_test.go once the files have been created.
2. And then no diagnostics once we fix the definition of DoIt.

What did you see instead?

1. No error diagnostics
2. Because we don't get the right error diagnostic state, the test stops so we can't verify this check

Marking as v0.3.0 because this is a fairly fundamental test case.

---

cc @stamblerre

FYI @leitzler

[stamblerre]

I tried reproducing this, but I wasn't able to get a failing test. Is this test expected to fail or will the logs just show the incorrect output?

[myitcv]

The test is not merged yet, but you can do:

git fetch origin pull/699/head && git checkout FETCH_HEAD

and then remove the skip from cmd/govim/testdata/scenario_default/quickfix_new_files.txt

Full repro steps using dockexec:

cd $(mktemp -d)
git clone https://github.com/govim/govim
cd govim
git fetch origin pull/699/head && git checkout FETCH_HEAD
sed -i '/skip/d' cmd/govim/testdata/scenario_default/quickfix_new_files.txt
go test -exec="dockexec govim/govim:go1.14beta1_vim_v8.2.0025_v1 -t -e VIM_FLAVOR=vim -e GOVIM_ERRLOGMATCH_WAIT=5s" -count=1 -v -run "TestScripts/scenario_default/quickfix_new_files" ./cmd/govim

gives

=== CONT  TestScripts/scenario_default/quickfix_new_files
    TestScripts/scenario_default/quickfix_new_files: testscript.go:348: 
        WORK=$WORK
        PATH=/vim/bin:/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
        HOME=$WORK/.home
        TMPDIR=$WORK/_tmp
        devnull=/dev/null
        :=:
        exe=
        GOPATH=$WORK/.home/gopath
        CCACHE_DISABLE=1
        GOARCH=amd64
        GOOS=linux
        GOROOT=/go
        GOCACHE=/gocache
        GOPROXY=http://127.0.0.1:34861/mod
        goversion=1.14
        GOVIM_ERRLOGMATCH_WAIT=5s
        GOVIM_DETECT_USER_BUSY=false
        GONOSUMDB=*
        PLUGIN_PATH=/start
        GOVIMTEST_SOCKET=127.0.0.1:37849
        GOVIMTESTDRIVER_SOCKET=127.0.0.1:42901
        
        # Test that we get diagnostics for new files that don't exist on disk.
        #
        # The scenario involves creating a package p with a simple function DoIt. p is
        # imported by a main package. p also has a test file that exercises DoIt, and
        # an external test file that does the same.
        #
        # Initially all call sites for DoIt incorrectly pass an integer argument,
        # meaning we should have error diagnostics for all call sites. Then we correct
        # the definition of DoIt to take an integer argument at which point all
        # diagnostics should disappear.
        # Create all the new files (0.109s)
        > vim ex 'e p/p.go'
        > vim ex 'r p/p.go.orig | 0d_'
        > vim ex 'noau w! check'
        > cmp check p/p.go.orig
        > vim ex 'sp main.go'
        > vim ex 'r main.go.orig | 0d_'
        > vim ex 'noau w! check'
        > cmp check main.go.orig
        > vim ex 'sp p/p_test.go'
        > vim ex 'r p/p_test.go.orig | 0d_'
        > vim ex 'noau w! check'
        > cmp check p/p_test.go.orig
        > vim ex 'sp p/x_test.go'
        > vim ex 'r p/x_test.go.orig | 0d_'
        > vim ex 'noau w! check'
        > cmp check p/x_test.go.orig
        # Expect the errors (4.942s)
        > vimexprwait errors.golden getqflist()
        [stderr]
        -[]
        +[
        +  {
        +    "bufnr": 1,
        +    "col": 9,
        +    "lnum": 6,
        +    "module": "",
        +    "nr": 0,
        +    "pattern": "",
        +    "text": "cannot convert 5 (untyped int constant) to string",
        +    "type": "",
        +    "valid": 1,
        +    "vcol": 0
        +  },
        +  {
        +    "bufnr": 2,
        +    "col": 7,
        +    "lnum": 6,
        +    "module": "",
        +    "nr": 0,
        +    "pattern": "",
        +    "text": "cannot convert 5 (untyped int constant) to string",
        +    "type": "",
        +    "valid": 1,
        +    "vcol": 0
        +  },
        +  {
        +    "bufnr": 3,
        +    "col": 9,
        +    "lnum": 9,
        +    "module": "",
        +    "nr": 0,
        +    "pattern": "",
        +    "text": "cannot convert 5 (untyped int constant) to string",
        +    "type": "",
        +    "valid": 1,
        +    "vcol": 0
        +  }
        +]
        [exit status 1]
        FAIL: testdata/scenario_default/quickfix_new_files.txt:32: unexpected command failure
        
--- FAIL: TestScripts (5.32s)
    --- FAIL: TestScripts/scenario_default (0.04s)
        --- FAIL: TestScripts/scenario_default/quickfix_new_files (5.31s)
FAIL

[stamblerre]

Thanks for the repro! I'll look into it ASAP.

[myitcv]

The test is now merged so repro steps are:

cd $(mktemp -d)
git clone https://github.com/govim/govim
cd govim
git checkout 1eb9311cf6155f0aceb9f9bb93dce20e406ec205
sed -i '/skip/d' cmd/govim/testdata/scenario_default/quickfix_new_files.txt
go test -exec="dockexec govim/govim:go1.14beta1_vim_v8.2.0025_v1 -t -e VIM_FLAVOR=vim -e GOVIM_ERRLOGMATCH_WAIT=5s" -count=1 -v -run "TestScripts/scenario_default/quickfix_new_files" ./cmd/govim

[heschi]

The problem here is that the initial workspace load finds nothing, and AFAIK we currently have no mechanism for adding packages to the workspace set, so they don't ever end up getting diagnosed in the background.

[myitcv]

FWIW this will be a common situation for Vim (and other non-VSCode) developers where the editor has an in-memory buffer that can exist without the file needing to exist on disk.