macOS ASAN builder is consistently failing

[itamaro]

These 3 tests consistently fail on the macOS ASAN builder

3 tests failed:
    test_cmd_line test_posix test_subprocess

Failing stacktraces:

``` 0:07:40 load avg: 8.91 Re-running 3 failed tests in verbose mode in subprocesses 0:07:40 load avg: 8.91 Run 3 tests in parallel using 3 worker processes (timeout: 20 min, worker timeout: 25 min) 0:07:41 load avg: 8.91 [1/3/1] test_posix failed (2 errors) Re-running test_posix in verbose mode (matching: test_close_file, test_close_file) test_close_file (test.test_posix.TestPosixSpawn.test_close_file) ... ERROR test_close_file (test.test_posix.TestPosixSpawnP.test_close_file) ... ERROR ====================================================================== ERROR: test_close_file (test.test_posix.TestPosixSpawn.test_close_file) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_posix.py", line 2008, in test_close_file with open(closefile, encoding="utf-8") as f: ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '@test_20686_tmpæ' ====================================================================== ERROR: test_close_file (test.test_posix.TestPosixSpawnP.test_close_file) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_posix.py", line 2008, in test_close_file with open(closefile, encoding="utf-8") as f: ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '@test_20686_tmpæ' ---------------------------------------------------------------------- Ran 2 tests in 0.351s FAILED (errors=2) test test_posix failed 0:07:41 load avg: 8.91 [2/3/2] test_subprocess failed (1 failure) Re-running test_subprocess in verbose mode (matching: test_pass_fds_redirected) test_pass_fds_redirected (test.test_subprocess.POSIXProcessTestCase.test_pass_fds_redirected) Regression test for https://bugs.python.org/issue32270. ... FAIL ====================================================================== FAIL: test_pass_fds_redirected (test.test_subprocess.POSIXProcessTestCase.test_pass_fds_redirected) Regression test for https://bugs.python.org/issue32270. ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_subprocess.py", line 3051, in test_pass_fds_redirected self.assertEqual(fds, {0, 1, 2} | frozenset(pass_fds), f"output={output!a}") ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Items in the first set but not the second: 3 : output=b'0,1,2,3,4,5,7\n' ---------------------------------------------------------------------- Ran 1 test in 0.285s FAILED (failures=1) test test_subprocess failed 0:07:41 load avg: 8.91 [3/3/3] test_cmd_line failed (4 failures) Re-running test_cmd_line in verbose mode (matching: test_no_std_streams, test_no_stderr, test_no_stdin, test_no_stdout) test_no_std_streams (test.test_cmd_line.CmdLineTest.test_no_std_streams) ... FAIL test_no_stderr (test.test_cmd_line.CmdLineTest.test_no_stderr) ... FAIL test_no_stdin (test.test_cmd_line.CmdLineTest.test_no_stdin) ... FAIL test_no_stdout (test.test_cmd_line.CmdLineTest.test_no_stdout) ... FAIL ====================================================================== FAIL: test_no_std_streams (test.test_cmd_line.CmdLineTest.test_no_std_streams) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_cmd_line.py", line 532, in test_no_std_streams self._test_no_stdio(['stdin', 'stdout', 'stderr']) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_cmd_line.py", line 520, in _test_no_stdio self.assertEqual(p.returncode, 42) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ AssertionError: 1 != 42 ====================================================================== FAIL: test_no_stderr (test.test_cmd_line.CmdLineTest.test_no_stderr) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_cmd_line.py", line 529, in test_no_stderr self._test_no_stdio(['stderr']) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^ File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_cmd_line.py", line 520, in _test_no_stdio self.assertEqual(p.returncode, 42) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ AssertionError: 1 != 42 ====================================================================== FAIL: test_no_stdin (test.test_cmd_line.CmdLineTest.test_no_stdin) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_cmd_line.py", line 523, in test_no_stdin self._test_no_stdio(['stdin']) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_cmd_line.py", line 520, in _test_no_stdio self.assertEqual(p.returncode, 42) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ AssertionError: 1 != 42 ====================================================================== FAIL: test_no_stdout (test.test_cmd_line.CmdLineTest.test_no_stdout) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_cmd_line.py", line 526, in test_no_stdout self._test_no_stdio(['stdout']) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^ File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_cmd_line.py", line 520, in _test_no_stdio self.assertEqual(p.returncode, 42) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ AssertionError: 1 != 42 ---------------------------------------------------------------------- Ran 4 tests in 0.772s FAILED (failures=4) test test_cmd_line failed 3 tests failed again: test_cmd_line test_posix test_subprocess

</details>

Example build: https://buildbot.python.org/all/#/builders/1353/builds/425

[itamaro]

I couldn't repro these failures on my MacBook, but from local debugging on the buildbot worker itself, I noticed this spamming the logs, which might be related:

python.exe(74602,0x7ff84626a700) malloc: nano zone abandoned due to inability to reserve vm space.

[itamaro]

Googling around, trying to set the env var MallocNanoZone to 0 is a possible workaround (or fix? idk).

This caused the failing tests to pass locally on the worker, so I'm going to try to set it in the builder config.

[vstinner]

malloc: nano zone abandoned due to inability to reserve vm space.

What does this message mean? Is it a memory allocation failure?

[itamaro]

malloc: nano zone abandoned due to inability to reserve vm space.

What does this message mean? Is it a memory allocation failure?

I'm not sure, I couldn't find any authoritative information on it. This SO thread has some useful comments.

[vstinner]

@itamaro created PR #451

[vstinner]

Thanks for the fix :-)