[dev.fuzz] testing: errors in (*testing.F).Cleanup functions do not always report the failure

[katiehockman]

When running go test -fuzz with -run=None and without setting -v, any errors/panics/failures inside of an f.Cleanup function are not reported.

This is likely related to #46631, as I discovered this issue while debugging that one. It appears we are handling f.Cleanup functions incorrectly when fuzzing right now. Possibly also related to changes from #46632.

Update: From trying to repro locally, sometimes it does the right thing, sometimes it doesn't, so there's probably a race somewhere.

Reproducer target:

func FuzzCleanupError(f *testing.F) {
	f.Cleanup(func() {
		f.Errorf("bad thing here")
	})
	f.Fuzz(func(t *testing.T, data []byte) {
		time.Sleep(time.Second)
	})
}

If run with gotip test -run=None -fuzz=FuzzCleanupError for a few seconds, then pressing Ctrl+C, the console doesn't print anything at all (neither an "ok" or a failure).

If the target is executed with -run, then it does print the failure correctly:

~ gotip test -run=FuzzCleanupError           
--- FAIL: FuzzCleanupError (0.00s)
    foo_test.go:46: bad thing here
FAIL
FAIL    foo   0.138s
FAIL

The same correctly reported failure occurs if run with gotip test -run=None -fuzz=FuzzCleanupError -v

/cc @golang/fuzzing

[jayconrod]

I wonder if the race here is happening because SIGINT is delivered to the coordinator and all the workers concurrently? So the coordinator could receive that signal first and shut down all the workers gracefully. Or a worker could receive it first and hit this error.

We could potentially make this more predictable by putting the coordinator and all workers in a new progress group. Then only the go test parent process would get SIGINT, and it would have to tell the coordinator to stop, and the coordinator would tell all the workers. I think that's roughly how it works on Windows; the shell sends CTRL_CLOSE_EVENT or something similar only to go test. I'm not sure we're handling that correctly either though.

[katiehockman]

This will likely be a release blocker for Go 1.18

[katiehockman]

I'm not able to reproduce this, anymore. I wouldn't be surprised if it got fixed along the way in our several refactors. I'm going to close this issue, but we can re-open it if users run into it.