Investigate flaky tests: test_wait_signal, test_wait, test_execve

[kamalmarhubi]

I've seen these fail occasionally

• sys::test_wait::test_wait_signal
• test_unistd::test_wait
• test_unistd::test_execve

Maybe some others

[kamalmarhubi]

Including output from @viraptor's report in #260:

test test_unistd::test_execve ... FAILED
test test_unistd::test_wait ... FAILED

failures:

---- test_unistd::test_execve stdout ----
        thread 'test_unistd::test_execve' panicked at 'called `Result::unwrap()` on an `Err` value: Sys(ECHILD)', ../src/libcore/result.rs:741

---- test_unistd::test_wait stdout ----
        thread 'test_unistd::test_wait' panicked at 'assertion failed: `(left == right)` (left: `Ok(Exited(2837, 0))`, right: `Ok(Exited(2843, 0))`)', test/test_unistd.rs:41


failures:
    test_unistd::test_execve
    test_unistd::test_wait

[kamalmarhubi]

new flaky test:

test sys::signal::tests::test_sigwait ... Process didn't exit successfully: `/Users/travis/build/kamalmarhubi/nix-rust/target/x86_64-apple-darwin/debug/nix-3f7e4935a20dbf98` (signal: 30)

[kamalmarhubi]

@posborne noted in #292:

I was mostly looking at the test_unistd failures which make calls out
to fork() and then make subsequent calls to wait(). In that case there
is one parent and the wait() called could (and frequently does) get some
random child pid back because it just happened to terminate. That is
why when one of the test fails so does the other one.

[posborne]

test_sigwait still seems to fail sometimes even with the single threaded execution. Must be a separate problem.

[kamalmarhubi]

Thoughts on these issues:

I was mostly looking at the test_unistd failures which make calls out
to fork() and then make subsequent calls to wait(). In that case there
is one parent and the wait() called could (and frequently does) get some
random child pid back because it just happened to terminate.

I'm going to make them wait on the child they spawn instead of any old child. I'm still not sure what is going on in test_sigwait. I couldn't see any failures on my linux machine when running single-threaded.

[dhylands]

Those errors (mentioned on Jan 31) look like the ones I saw as well.

[kamalmarhubi]

@posborne

Calling the actual system calls only lets us test the happy path (and even then not always as consistently as we would like).

I have some ideas on this as well, but I'll check your link. I think it's a separate issue from the flaky / intermittent failures though.

[dhylands]

I'll bet its race where the SIGUSR1 is being delivered before the wait is called. The error:

build/kamalmarhubi/nix-rust/target/x86_64-apple-darwin/debug/nix-3f7e4935a20dbf98` (signal: 30)

seems to be saying that the process was killed by SIGUSR1. If a context switch happened after the raise and before the call to wait, then it's entirely possible that that signal gets processed before thewait, and since no one is yet waiting for the signal, it terminates the process.

But I'm really unfamiliar with the OSX kernel stuff, so I could be spouting nonsense.

[kamalmarhubi]

I opened #303 for the more general how-to-test issue.