os/signal: TestAtomicStop failing on Illumos

[bradfitz]

The newly revived Illumos builder (run by @jclulow now, on different host/OS probably) is now failing with:

https://build.golang.org/log/47c0329f33a7c7bd68dc98c35021160b03a3c6a5

ok  	os	0.809s
ok  	os/exec	0.869s
--- FAIL: TestAtomicStop (2.15s)
    signal_test.go:400: iteration 1: output lost signal on tries: 2
    signal_test.go:408: iteration 1: lost signal
FAIL
FAIL	os/signal	6.493s
ok  	os/user	0.020s
ok  	path	0.025s
FAIL
2019/10/22 14:36:24 Failed: exit status 1

/cc @ianlancetaylor @bcmills

[jclulow]

I was just looking at this. It doesn't, as far as I can tell, always fail -- just most of the time. Having watched the buildlet a bit in the last few days while it was getting going, the load averages in the zone get up pretty high.

I think this is because the -p flag for job parallelism defaults to the number of CPUs, so the test runner starts that many tests in parallel. Because this is a zone (i.e., a kind of container) you can actually see (and schedule on!) every CPU in the host; there are 48 in all. Though all of the CPUs are visible, the zone is capped in the amount of CPU time it is able to use -- in effect, it's able to use 2 CPUs worth of CPU time per second.

If TestAtomicStop is timing sensitive, it's possible that when it fails it's running alongside rather a lot of other tests, and the ~24X over subscription might mean this test isn't on CPU to run soon enough to hit the timing window it's expecting. When I run the test suite on a standalone machine which doesn't have this property (i.e., a small system on my desk where I'm the only user) it seems to pass every time.

[bradfitz]

On Linux, runtime.NumCPU() looks at the number of actually usable CPUs.

Perhaps the fix here is just to fix the Illumos implementation of NumCPU to be aware of the zone's CPU limit.

[bradfitz]

In fact, I dealt with this just recently in https://go-review.googlesource.com/c/build/+/201637/7/cmd/rundockerbuildlet/rundockerbuildlet.go#162 where I had to restrict a Docker container's --cpuset-cpus value just to make runtime.NumCPU happy, even though it would've been nice to give the containers as much extra CPU as they wanted to fight over.

Perhaps we need a new knob that lets us control test parallelism without hard-coding it to mean NumCPU.

/cc @bcmills

[jclulow]

If we could set, say GO_BUILD_MAXJOBS, and have that override the -p default, I could definitely set that in the environment for the buildlet service to a value that makes sense for the specific memory and CPU caps in the particular zone.

[bradfitz]

@jclulow, you should probably still fix runtime.NumCPU meanwhile, though, if it's not too much work.

[bcmills]

The hard-coded time.After in this test is suspect:

go/src/os/signal/signal_test.go

Lines 449 to 456 in 606019c

	// At this point we should either die from SIGINT or
	// get a notification on cs. If neither happens, we
	// dropped the signal. It is given 2 seconds to
	// deliver, as needed for gccgo on some loaded test systems.
	select {
	case <-cs:
	case <-time.After(2 * time.Second):

See previously #29046.

[bcmills]

I don't see a compelling need to limit the delay here to 2 seconds, given that in the normal case the test will pass on every attempt.

[laboger]

FWIW, we see this failure intermittently on gccgo for both ppc64le & ppc64. The limit was put at 2s to hopefully fix that problem but it still happens. We don't see the error in golang.

[bcmills]

If we pull in the fix for #28135, then we could fairly easily plumb the test deadline down to atomicStopTestProgram and remove the hard-coded timeout entirely. I'm not sure whether that would address the problem for gccgo or make the failure mode more frustrating, though.

[gopherbot]

Change https://golang.org/cl/203499 mentions this issue: os/signal: derive TestAtomicStop timeout from overall test timeout