runtime: unresponsive tests with a `running on other thread` goroutine created by `runtime.gcenable`

[gopherbot]

#!watchflakes
post <- log ~ `Test killed with quit` && log ~ `goroutine \d+ \[running\]:\n\s+goroutine running on other thread; stack unavailable\n\s*created by runtime\.gcenable`

Issue created automatically to collect these failures.

Example (log):

SIGQUIT: quit
PC=0x873fc m=0 sigcode=0

goroutine 2 [running]:
runtime.systemstack_switch()
	/workdir/go/src/runtime/asm_ppc64x.s:212 +0x10 fp=0xc00003cd70 sp=0xc00003cd50 pc=0x83180
runtime.(*mheap).freeSpan(...)
	/workdir/go/src/runtime/mheap.go:1551
runtime.(*sweepLocked).sweep(0x5a02c0?, 0x0)
	/workdir/go/src/runtime/mgcsweep.go:757 +0x73c fp=0xc00003ce78 sp=0xc00003cd70 pc=0x396fc
...
r20  0x54	r21  0x584680
r22  0x7ffff3f4ab70	r23  0xc00003cdf0
r24  0x561a40	r25  0x16
r26  0x0	r27  0x0
r28  0x0	r29  0xc000030258
r30  0x583a40	r31  0x1bee0
pc   0x873fc	ctr  0x0
link 0x48b50	xer  0x0
ccr  0x34448084	trap 0xc00
*** Test killed with quit: ran too long (22m0s).

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "golang.org/x/tools/internal/gcimporter" && test == ""

2023-11-09 21:01 linux-ppc64le-power10osu tools@e3f67986 go@130baf3d x/tools/internal/gcimporter (log)

SIGQUIT: quit
PC=0x873fc m=0 sigcode=0

goroutine 2 [running]:
runtime.systemstack_switch()
	/workdir/go/src/runtime/asm_ppc64x.s:212 +0x10 fp=0xc00003cd70 sp=0xc00003cd50 pc=0x83180
runtime.(*mheap).freeSpan(...)
	/workdir/go/src/runtime/mheap.go:1551
runtime.(*sweepLocked).sweep(0x5a02c0?, 0x0)
	/workdir/go/src/runtime/mgcsweep.go:757 +0x73c fp=0xc00003ce78 sp=0xc00003cd70 pc=0x396fc
...
r20  0x54	r21  0x584680
r22  0x7ffff3f4ab70	r23  0xc00003cdf0
r24  0x561a40	r25  0x16
r26  0x0	r27  0x0
r28  0x0	r29  0xc000030258
r30  0x583a40	r31  0x1bee0
pc   0x873fc	ctr  0x0
link 0x48b50	xer  0x0
ccr  0x34448084	trap 0xc00
*** Test killed with quit: ran too long (22m0s).

2023-11-10 13:59 linux-ppc64le-power10osu tools@92a8009c go@abf84221 x/tools/internal/gcimporter (log)

SIGQUIT: quit
PC=0x875bc m=0 sigcode=0

goroutine 2 [running]:
	goroutine running on other thread; stack unavailable
created by runtime.init.5 in goroutine 1
	/workdir/go/src/runtime/proc.go:314 +0x2c

goroutine 3 [running]:
	goroutine running on other thread; stack unavailable
...
r20  0xaa	r21  0x5846e0
r22  0x7fffe31f23a0	r23  0xc000d42b10
r24  0xc000771930	r25  0xc000771908
r26  0xc003100c39	r27  0xc003100c35
r28  0x0	r29  0xc000032e98
r30  0x583aa0	r31  0x1c2b0
pc   0x875bc	ctr  0x0
link 0x48bd0	xer  0x0
ccr  0x34448024	trap 0xc00
*** Test killed with quit: ran too long (22m0s).

— watchflakes

[bcmills]

@golang/runtime: this looks to me like a runtime deadlock or livelock. Notably, the test binary's internal timer failed to fire.

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `Test killed with quit` && log ~ `goroutine \d+ \[running\]:\n\s*runtime\.systemstack_switch`

2023-10-02 15:18 aix-ppc64 go@c9c885f9 os/exec.TestCredentialNoSetGroups (log)

SIGQUIT: quit
PC=0x900000000216b38 m=9 sigcode=0

r0   0xffffffffffffffff	r1   0x11049d130
r2   0xffffffffffffffff	r3   0xffffffffffffffff
r4   0xffffffffffffffff	r5   0xffffffffffffffff
r6   0xffffffffffffffff	r7   0xffffffffffffffff
r8   0xffffffffffffffff	r9   0xffffffffffffffff
r10  0xffffffffffffffff	r11  0xffffffffffffffff
r12  0xffffffffffffffff	r13  0x1104a5800
...
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/os/user/lookup.go:22 +0x24 fp=0xa00010000200da0 sp=0xa00010000200d80 pc=0x10015cba4
sync.(*Once).doSlow(0xa00010000003d40?, 0x1002f009b?)
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/sync/once.go:74 +0x124 fp=0xa00010000200e08 sp=0xa00010000200da0 pc=0x100086cb4
sync.(*Once).Do(...)
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/sync/once.go:65
os/user.Current()
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/os/user/lookup.go:22 +0x54 fp=0xa00010000200e68 sp=0xa00010000200e08 pc=0x10015c7c4
os/exec_test.TestCredentialNoSetGroups(0xa000100001c5380)
	/ramdisk8GB/workdir-host-aix-ppc64-osuosl/go/src/os/exec/exec_posix_test.go:45 +0x38 fp=0xa00010000200f30 sp=0xa00010000200e68 pc=0x1002d4e48
testing.tRunner(0xa000100001c5380, 0x1100902c8)

— watchflakes

[bcmills]

Huh. Yeah, my attempt at a watchflakes pattern is clearly not right — it's catching #56418 but not all the failures for this issue. 🙃

[mknyszek]

I believe this is the same issue as #64050. Working on it now.

[mknyszek]

Wait, actually I see the issue here very clearly in that stack trace. ... Huh. This seems unrelated to basically anything else that has landed recently, but I'm positive it's the problem.