Skip to content

macOS ASAN builder is consistently failing #450

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
itamaro opened this issue Jan 15, 2024 · 6 comments
Closed

macOS ASAN builder is consistently failing #450

itamaro opened this issue Jan 15, 2024 · 6 comments

Comments

@itamaro
Copy link
Contributor

itamaro commented Jan 15, 2024

These 3 tests consistently fail on the macOS ASAN builder

3 tests failed:
    test_cmd_line test_posix test_subprocess

Failing stacktraces:

``` 0:07:40 load avg: 8.91 Re-running 3 failed tests in verbose mode in subprocesses 0:07:40 load avg: 8.91 Run 3 tests in parallel using 3 worker processes (timeout: 20 min, worker timeout: 25 min) 0:07:41 load avg: 8.91 [1/3/1] test_posix failed (2 errors) Re-running test_posix in verbose mode (matching: test_close_file, test_close_file) test_close_file (test.test_posix.TestPosixSpawn.test_close_file) ... ERROR test_close_file (test.test_posix.TestPosixSpawnP.test_close_file) ... ERROR ====================================================================== ERROR: test_close_file (test.test_posix.TestPosixSpawn.test_close_file) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_posix.py", line 2008, in test_close_file with open(closefile, encoding="utf-8") as f: ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '@test_20686_tmpæ' ====================================================================== ERROR: test_close_file (test.test_posix.TestPosixSpawnP.test_close_file) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_posix.py", line 2008, in test_close_file with open(closefile, encoding="utf-8") as f: ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '@test_20686_tmpæ' ---------------------------------------------------------------------- Ran 2 tests in 0.351s FAILED (errors=2) test test_posix failed 0:07:41 load avg: 8.91 [2/3/2] test_subprocess failed (1 failure) Re-running test_subprocess in verbose mode (matching: test_pass_fds_redirected) test_pass_fds_redirected (test.test_subprocess.POSIXProcessTestCase.test_pass_fds_redirected) Regression test for https://bugs.python.org/issue32270. ... FAIL ====================================================================== FAIL: test_pass_fds_redirected (test.test_subprocess.POSIXProcessTestCase.test_pass_fds_redirected) Regression test for https://bugs.python.org/issue32270. ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_subprocess.py", line 3051, in test_pass_fds_redirected self.assertEqual(fds, {0, 1, 2} | frozenset(pass_fds), f"output={output!a}") ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Items in the first set but not the second: 3 : output=b'0,1,2,3,4,5,7\n' ---------------------------------------------------------------------- Ran 1 test in 0.285s FAILED (failures=1) test test_subprocess failed 0:07:41 load avg: 8.91 [3/3/3] test_cmd_line failed (4 failures) Re-running test_cmd_line in verbose mode (matching: test_no_std_streams, test_no_stderr, test_no_stdin, test_no_stdout) test_no_std_streams (test.test_cmd_line.CmdLineTest.test_no_std_streams) ... FAIL test_no_stderr (test.test_cmd_line.CmdLineTest.test_no_stderr) ... FAIL test_no_stdin (test.test_cmd_line.CmdLineTest.test_no_stdin) ... FAIL test_no_stdout (test.test_cmd_line.CmdLineTest.test_no_stdout) ... FAIL ====================================================================== FAIL: test_no_std_streams (test.test_cmd_line.CmdLineTest.test_no_std_streams) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_cmd_line.py", line 532, in test_no_std_streams self._test_no_stdio(['stdin', 'stdout', 'stderr']) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_cmd_line.py", line 520, in _test_no_stdio self.assertEqual(p.returncode, 42) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ AssertionError: 1 != 42 ====================================================================== FAIL: test_no_stderr (test.test_cmd_line.CmdLineTest.test_no_stderr) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_cmd_line.py", line 529, in test_no_stderr self._test_no_stdio(['stderr']) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^ File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_cmd_line.py", line 520, in _test_no_stdio self.assertEqual(p.returncode, 42) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ AssertionError: 1 != 42 ====================================================================== FAIL: test_no_stdin (test.test_cmd_line.CmdLineTest.test_no_stdin) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_cmd_line.py", line 523, in test_no_stdin self._test_no_stdio(['stdin']) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_cmd_line.py", line 520, in _test_no_stdio self.assertEqual(p.returncode, 42) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ AssertionError: 1 != 42 ====================================================================== FAIL: test_no_stdout (test.test_cmd_line.CmdLineTest.test_no_stdout) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_cmd_line.py", line 526, in test_no_stdout self._test_no_stdio(['stdout']) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^ File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.asan/build/Lib/test/test_cmd_line.py", line 520, in _test_no_stdio self.assertEqual(p.returncode, 42) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ AssertionError: 1 != 42 ---------------------------------------------------------------------- Ran 4 tests in 0.772s FAILED (failures=4) test test_cmd_line failed 3 tests failed again: test_cmd_line test_posix test_subprocess
</details>

Example build: https://buildbot.python.org/all/#/builders/1353/builds/425
@itamaro
Copy link
Contributor Author

itamaro commented Jan 15, 2024

I couldn't repro these failures on my MacBook, but from local debugging on the buildbot worker itself, I noticed this spamming the logs, which might be related:

python.exe(74602,0x7ff84626a700) malloc: nano zone abandoned due to inability to reserve vm space.

@itamaro
Copy link
Contributor Author

itamaro commented Jan 15, 2024

Googling around, trying to set the env var MallocNanoZone to 0 is a possible workaround (or fix? idk).

This caused the failing tests to pass locally on the worker, so I'm going to try to set it in the builder config.

@vstinner
Copy link
Member

malloc: nano zone abandoned due to inability to reserve vm space.

What does this message mean? Is it a memory allocation failure?

@itamaro
Copy link
Contributor Author

itamaro commented Jan 16, 2024

malloc: nano zone abandoned due to inability to reserve vm space.

What does this message mean? Is it a memory allocation failure?

I'm not sure, I couldn't find any authoritative information on it. This SO thread has some useful comments.

@vstinner
Copy link
Member

@itamaro created PR #451

@vstinner
Copy link
Member

Thanks for the fix :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants