runtime: timeout in TestNeedmDeadlock on darwin-amd64-10_12 builder

[bcmills]

2021-02-15T01:54:22-33d72fd/darwin-amd64-10_12
2020-10-29T15:13:09-50af50d/darwin-amd64-10_12

It looks like the testprogcgo process ends up wedged, but runBuildTestProg fails to terminate it so we don't get a useful stack trace.

See previously #42207 (the deadlock for which this test was added).
See also #43625 (TestNeedmDeadlock on MIPS).

goroutine 22881 [syscall, 2 minutes]:
syscall.syscall6(0x108d5e0, 0x19ef, 0xc0003bec74, 0x0, 0xc00007a090, 0x0, 0x0, 0x0, 0x0, 0x0)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/runtime/sys_darwin.go:41 +0x2e fp=0xc0003bebc0 sp=0xc0003beb98 pc=0x107822e
syscall.wait4(0x19ef, 0xc0003bec74, 0x0, 0xc00007a090, 0x90, 0x1285a20, 0xc0003bed01)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/syscall/zsyscall_darwin_amd64.go:44 +0x85 fp=0xc0003bec38 sp=0xc0003bebc0 pc=0x108a345
syscall.Wait4(0x19ef, 0xc0003becc4, 0x0, 0xc00007a090, 0xc00010e420, 0x60, 0x100fd7b)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/syscall/syscall_bsd.go:144 +0x51 fp=0xc0003bec88 sp=0xc0003bec38 pc=0x1088af1
os.(*Process).wait(0xc000078030, 0xc00010e420, 0x12a5560, 0xc000080480)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/os/exec_unix.go:43 +0x85 fp=0xc0003becf8 sp=0xc0003bec88 pc=0x10a48c5
os.(*Process).Wait(...)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/os/exec.go:129
os/exec.(*Cmd).Wait(0xc0003c8000, 0x12a7ab8, 0xc00010e420)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/os/exec/exec.go:507 +0x65 fp=0xc0003bed78 sp=0xc0003becf8 pc=0x113b785
runtime_test.runBuiltTestProg(0xc000483500, 0xc00013e100, 0x71, 0x1291263, 0xd, 0x0, 0x0, 0x0, 0x0, 0x0)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/runtime/crash_test.go:103 +0x2ed fp=0xc0003bee70 sp=0xc0003bed78 pc=0x11810cd
runtime_test.runTestProg(0xc000483500, 0x1290916, 0xb, 0x1291263, 0xd, 0x0, 0x0, 0x0, 0xc0321263d0, 0x321263d0010f64d0)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/runtime/crash_test.go:57 +0x165 fp=0xc0003beef0 sp=0xc0003bee70 pc=0x1180d45
runtime_test.TestNeedmDeadlock(0xc000483500)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/runtime/crash_cgo_test.go:628 +0x7b fp=0xc0003bef80 sp=0xc0003beef0 pc=0x11809fb
testing.tRunner(0xc000483500, 0x12a7170)
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/testing/testing.go:1194 +0xef fp=0xc0003befd0 sp=0xc0003bef80 pc=0x10f6b2f
runtime.goexit()
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0003befd8 sp=0xc0003befd0 pc=0x107b421
created by testing.(*T).Run
	/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/src/testing/testing.go:1239 +0x2b3
FAIL	runtime	180.025s

CC @ianlancetaylor @cherrymui

[bcmills]

Tentatively milestoning as a release-blocker for Go 1.17 until we better understand the root cause. This may be a deadlock in the runtime, and the failure mode would make it very difficult for a Go user to debug.

[dmitshur]

CC @prattmic.

[dmitshur]

Friendly ping after 2 weeks since it's a release blocking issue for Go 1.17.

[ianlancetaylor]

There seem to only be two failures since the test was added in October, 2020. Both failures are on Darwin 10.12. In both failures the test hits a 3 minute timeout, which is interesting because

1. The test program (in runtime/testdata/testprogcgo/needmdeadlock.go) itself has a 5 second timeout before exiting, and
2. the test runner (runBuiltTestProg in runtime/crash_test.go) has a 1 minute timeout before sending a SIGQUIT signal.

It seems that the test program has deadlocked so hard that both of those timeouts fail to stop it.

[ianlancetaylor]

This has only happened twice, it's only happened on macOS 10.12, and we are dropping support for macOS 10.12 in 1.17 anyhow. Removing the release-blocker label.

[gopherbot]

Change https://golang.org/cl/312692 mentions this issue: dashboard: limit macOS 10.12 builders to at most go1.16

[dmitshur]

macOS 10.12 is no longer supported by any supported Go release, and the builder was dropped in CL 419194. Let's close this @bcmills?