[dev.fuzz] cmd/go: hangs in "gathering baseline coverage"

[klauspost]

What version of Go are you using (go version)?

λ gotip version
go version devel go1.17-5542c10fbf Fri Jun 4 03:07:33 2021 +0000 windows/amd64

What operating system and processor architecture are you using (go env)?

go env Output

λ gotip env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=d:\temp\gocache
set GOENV=C:\Users\klaus\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=e:\gopath\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=e:\gopath
set GOPRIVATE=
set GOPROXY=https://goproxy.io
set GOROOT=C:\Users\klaus\sdk\gotip
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Users\klaus\sdk\gotip\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=devel go1.17-5542c10fbf Fri Jun 4 03:07:33 2021 +0000
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=NUL
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=d:\temp\wintemp\go-build1674875180=/tmp/go-build -gno-record-gcc-switches

What did you do?

Run fuzz test:

func FuzzMinimize(f *testing.F) {
	f.Add([]byte{4: 0})
	f.Fuzz(func(t *testing.T, data []byte) {
		if len(data) < 4 {
			return
		}
		if bits.OnesCount32(binary.LittleEndian.Uint32(data)) >= 28 {
			t.Fatal("err")
		}
	})
}

First output generates crash. Generates file in GOCACHE: 7777bc1fe491b6e2c55af04cf305ec8b045058c3c61be9eaae8be7fb0859e4c3.gz

(gzipped for github)

λ gotip test -fuzz=FuzzMinimize -test.run=None                                                                                   
found a crash, minimizing...                                                                                                     
fuzzing, elapsed: 1.2s, execs: 7 (6/sec), workers: 32, interesting: 1                                                            
--- FAIL: FuzzMinimize (1.17s)                                                                                                   
        --- FAIL: FuzzMinimize (0.00s)                                                                                           
            fuzz_test.go:227: err                                                                                                
                                                                                                                                 
    Crash written to testdata\corpus\FuzzMinimize\1770021c52f7a4641b23ff7c71ebe5fce49215cf4935a3d737487b5e598662d6               
    To re-run:                                                                                                                   
    go test github.com/klauspost/compress/zstd -run=FuzzMinimize/1770021c52f7a4641b23ff7c71ebe5fce49215cf4935a3d737487b5e598662d6
FAIL                                                                                                                             
exit status 1                                                                                                                    
FAIL    github.com/klauspost/compress/zstd      1.236s                                                                           

Restarting the fuzzer appears stuck:

λ gotip test -fuzz=FuzzMinimize -test.run=None
gathering baseline coverage, elapsed: 3.0s, workers: 32, left: 1
gathering baseline coverage, elapsed: 6.0s, workers: 32, left: 1
gathering baseline coverage, elapsed: 9.0s, workers: 32, left: 1
...
gathering baseline coverage, elapsed: 657.0s, workers: 32, left: 1
...

What did you expect to see?

Fuzzer would exit or progress.

What did you see instead?

Fuzzer stuck in "gathering baseline coverage" forever.

[klauspost]

Maybe inherited from dvyukov/go-fuzz#276

[katiehockman]

/cc @jayconrod

[dchapes]

	f.Fuzz(func(t *testing.T, data []byte) {
		if len(data) < 4 {
			return
		}

By the way, from my understanding of the new fuzz implementation, I believe you'd normally want to use t.SkipNow() (or t.Skip(…)) instead of a plain return here. The former tells the fuzz engine the input was uninteresting where-as the later tells it the input passed (I think; I'm in no way involved with the development of this fuzzer). Either way, the fuzzer shouldn't get stuck.

[rolandshoemaker]

I believe this is caused by using t.Fatal, which terminates the test runner before it can snapshot the coverage data, causing the worker to send a nil slice to the coordinator, which expects all of the initial corpus inputs to return coverage data. We can fix this by figuring out how to properly capture coverage when t.Fatal is used (or the function panics).

It's also caused by using -test.run=none, since that skips the initial testing of seeds and the worker doesn't inspect errors for initial coverage gathering. We may want to explicitly check those errors in order to avoid this particular pattern.

[rolandshoemaker]

Weirdly I don't see this with -parallel=1, and I'm not entirely sure why.

It's because once a good input is processed the coverage slice becomes non-nil with old coverage information, so the coordinator gets something it likes.

[gopherbot]

Change https://golang.org/cl/328650 mentions this issue: [dev.fuzz] testing: capture coverage even if tRunner failed

[katiehockman]

Sounds like this is fixed. Gopherbot just didn't realize because the Fixes line should have been Fixes golang/go#46633