test_binascii: test_b2a_roundtrip() failure with Hypothesis

[vstinner]

It seems like Hypothesis found an issue. Can someone have a look? How can it be reproduced locally?

ERROR: test_b2a_roundtrip (test.test_binascii.MemoryviewBinASCIITest.test_b2a_roundtrip)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_binascii.py", line 233, in test_b2a_roundtrip
    binary=hypothesis.strategies.binary(),
           ^^^^^^^
  File "/home/runner/work/cpython/cpython-builddir/hypovenv/lib/python3.13/site-packages/hypothesis/core.py", line 1386, in wrapped_test
    raise the_error_hypothesis_found
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_binascii.py", line 237, in test_b2a_roundtrip
    converted = binascii.b2a_uu(self.type2test(binary), backtick=backtick)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
binascii.Error: At most 45 bytes at once
Falsifying example: test_b2a_roundtrip(
    self=<test.test_binascii.MemoryviewBinASCIITest testMethod=test_b2a_roundtrip>,
    binary=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
    backtick=False,
)

You can reproduce this example by temporarily adding @reproduce_failure('6.84.0', b'AXicY2RgYBwciIEBAA2TAC8=') as a decorator on your test case

----------------------------------------------------------------------
Ran 85 tests in 127.421s

FAILED (errors=1, skipped=1)
test test_binascii failed

build: https://github.com/python/cpython/actions/runs/6189519251/job/16803785370

[vstinner]

I modified the test as requested to try to reproduce the issue:

diff --git a/Lib/test/test_binascii.py b/Lib/test/test_binascii.py
index 3d3e0746e9..cec72c5c78 100644
--- a/Lib/test/test_binascii.py
+++ b/Lib/test/test_binascii.py
@@ -5,7 +5,7 @@
 import array
 import re
 from test.support import bigmemtest, _1G, _4G
-from test.support.hypothesis_helper import hypothesis
+import hypothesis
 
 
 # Note: "*_hex" functions are aliases for "(un)hexlify"
@@ -16,6 +16,7 @@
 all_functions = a2b_functions + b2a_functions + ['crc32', 'crc_hqx']
 
 
+@hypothesis.reproduce_failure('6.84.0', b'AXicY2RgYBwciIEBAA2TAC8=')
 class BinASCIITest(unittest.TestCase):
 
     type2test = bytes

I install hypothesis in a venv:

./python -m venv env
env/bin/python -m pip install hypothesis

When I try to reproduce the failure, I get:

vstinner@mona$ env/bin/python -m test test_binascii -v
(...)
======================================================================
ERROR: test_b2a_qp_a2b_qp_round_trip (test.test_binascii.BinASCIITest.test_b2a_qp_a2b_qp_round_trip)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_binascii.py", line 400, in test_b2a_qp_a2b_qp_round_trip
    binary=hypothesis.strategies.binary(),
           ^^^^^^^
  File "/home/vstinner/python/main/env/lib/python3.13/site-packages/hypothesis/core.py", line 1263, in wrapped_test
    fail_health_check(settings, msg, HealthCheck.differing_executors)
  File "/home/vstinner/python/main/env/lib/python3.13/site-packages/hypothesis/internal/healthcheck.py", line 27, in fail_health_check
    raise FailedHealthCheck(message)
hypothesis.errors.FailedHealthCheck: The method BinASCIITest.test_b2a_qp_a2b_qp_round_trip was called from multiple different executors. This may lead to flaky tests and nonreproducible errors when replaying from database.
See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.differing_executors to the suppress_health_check settings for this test.
(...)

And then many tests fail with the same error message:

• test_b2a_qp_a2b_qp_round_trip (test.test_binascii.BinASCIITest.test_b2a_qp_a2b_qp_round_trip)
• test_b2a_roundtrip (test.test_binascii.BinASCIITest.test_b2a_roundtrip)
• test_base64_roundtrip (test.test_binascii.BinASCIITest.test_base64_roundtrip)
• test_hex_roundtrip (test.test_binascii.BinASCIITest.test_hex_roundtrip)
• test_b2a_qp_a2b_qp_round_trip (test.test_binascii.BytearrayBinASCIITest.test_b2a_qp_a2b_qp_round_trip)
• test_b2a_roundtrip (test.test_binascii.BytearrayBinASCIITest.test_b2a_roundtrip)
• test_base64_roundtrip (test.test_binascii.BytearrayBinASCIITest.test_base64_roundtrip)
• test_hex_roundtrip (test.test_binascii.BytearrayBinASCIITest.test_hex_roundtrip)
• test_b2a_qp_a2b_qp_round_trip (test.test_binascii.MemoryviewBinASCIITest.test_b2a_qp_a2b_qp_round_trip)
• test_b2a_roundtrip (test.test_binascii.MemoryviewBinASCIITest.test_b2a_roundtrip)
• test_base64_roundtrip (test.test_binascii.MemoryviewBinASCIITest.test_base64_roundtrip)
• test_hex_roundtrip (test.test_binascii.MemoryviewBinASCIITest.test_hex_roundtrip)

I gave up at the FailedHealthCheck.

[vstinner]

cc @pganssle @Zac-HD

[hugovk]

The good news is this is a duplicate of #109418 that was fixed moments ago by #109419.

Perhaps @sobolevn can assist with reproducing, if you're still interested?

[sobolevn]

@vstinner try this:

» git patch
diff --git Lib/test/test_binascii.py Lib/test/test_binascii.py
index ef744f6b972..db7d25ab98e 100644
--- Lib/test/test_binascii.py
+++ Lib/test/test_binascii.py
@@ -233,6 +233,7 @@ def test_uu(self):
         binary=hypothesis.strategies.binary(max_size=45),
         backtick=hypothesis.strategies.booleans(),
     )
+    @hypothesis.reproduce_failure('6.84.0', b'AXicY2RgYBwciIEBAA2TAC8=')
     def test_b2a_roundtrip(self, binary, backtick):
         converted = binascii.b2a_uu(self.type2test(binary), backtick=backtick)
         restored = binascii.a2b_uu(self.type2test(converted))
                                                               

It produces a correct result on main (after my fix):

======================================================================
ERROR: test_b2a_roundtrip (test.test_binascii.MemoryviewBinASCIITest.test_b2a_roundtrip)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_binascii.py", line 233, in test_b2a_roundtrip
    binary=hypothesis.strategies.binary(max_size=45),
           ^^^^^^^
  File "/Users/sobolev/Desktop/cpython/.venv/lib/python3.13/site-packages/hypothesis/core.py", line 1290, in wrapped_test
    raise DidNotReproduce(
hypothesis.errors.DidNotReproduce: Expected the test to raise an error, but it completed successfully.

----------------------------------------------------------------------

[sobolevn]

Btw, if you have any questions about hypothesis, feel free to ping me :)

[vstinner]

hypothesis.errors.FailedHealthCheck: The method BinASCIITest.test_b2a_qp_a2b_qp_round_trip was called from multiple different executors. This may lead to flaky tests and nonreproducible errors when replaying from database.
See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.differing_executors to the suppress_health_check settings for this test.

I don't understand this message. Why does it want to contact a remote server since I want to reproduce the issue locally? Maybe the hint in the first message is wrong?

You can reproduce this example by temporarily adding @reproduce_failure('6.84.0', b'AXicY2RgYBwciIEBAA2TAC8=') as a decorator on your test case

[vstinner]

The good news is this is a duplicate of #109418 that was fixed moments ago by #109419.

Oh, two hours ago, nice.

[sobolevn]

I don't understand this message. Why does it want to contact a remote server since I want to reproduce the issue locally? Maybe the hint in the first message is wrong?

This is interesting, it should not happen, because I disabled this check ~a week ago:

cpython/Lib/test/support/hypothesis_helper.py

Lines 13 to 16 in 68a6f21

	suppress_health_check=[
	hypothesis.HealthCheck.too_slow,
	hypothesis.HealthCheck.differing_executors,
	],

I will try to reproduce your case.