gh-108511: Add C API functions which do not silently ignore errors #109025

serhiy-storchaka · 2023-09-06T20:13:30Z

Add the following functions:

PyObject_HasAttrWithError()
PyObject_HasAttrStringWithError()
PyMapping_HasKeyWithError()
PyMapping_HasKeyStringWithError()

Issue: C API: Add replacements for PyObject_HasAttr() etc #108511

📚 Documentation preview 📚: https://cpython-previews--109025.org.readthedocs.build/

Add the following functions: * PyObject_HasAttrWithError() * PyObject_HasAttrStringWithError() * PyMapping_HasKeyWithError() * PyMapping_HasKeyStringWithError()

bedevere-bot · 2023-09-17T11:36:25Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL8 3.x has failed when building commit add16f1.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/509/builds/4924) and take a look at the build logs.
Check if the failure is related to this commit (add16f1) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/509/builds/4924

Failed tests:

test_tools

Failed subtests:

test_freeze_simple_script - test.test_tools.test_freeze.TestFreeze.test_freeze_simple_script

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/test/test_tools/test_freeze.py", line 28, in test_freeze_simple_script
    outdir, scriptfile, python = helper.prepare(script, outdir)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Tools/freeze/test/freeze.py", line 146, in prepare
    copy_source_tree(srcdir, SRCDIR)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Tools/freeze/test/freeze.py", line 95, in copy_source_tree
    shutil.copytree(oldroot, newroot, ignore=ignore_non_src)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/shutil.py", line 588, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/shutil.py", line 542, in _copytree
    raise Error(errors)
shutil.Error: [('/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/build/test_python_2730921æ/tmpiuadha4u', '/tmp/test_python_yi9eetzr/tmp8zxr_xn3/cpython/build/test_python_2730921æ/tmpiuadha4u', "[Errno 2] No such file or directory: '/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/build/test_python_2730921æ/tmpiuadha4u'")]

bedevere-bot · 2023-09-17T11:43:18Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 Debian Clang LTO + PGO 3.x has failed when building commit add16f1.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1084/builds/2016) and take a look at the build logs.
Check if the failure is related to this commit (add16f1) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1084/builds/2016

Failed tests:

test.test_multiprocessing_fork.test_misc

Failed subtests:

test_nested_startmethod - test.test_multiprocessing_fork.test_misc.TestStartMethod.test_nested_startmethod

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/var/lib/buildbot/workers/arm64-clang/3.x.gps-arm64-debian.clang.lto-pgo/build/Lib/test/_test_multiprocessing.py", line 5475, in test_nested_startmethod
    self.assertEqual(results, [2, 1])
AssertionError: Lists differ: [1, 2] != [2, 1]

bedevere-bot · 2023-09-17T11:50:14Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL7 LTO + PGO 3.x has failed when building commit add16f1.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/244/builds/5456) and take a look at the build logs.
Check if the failure is related to this commit (add16f1) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/244/builds/5456

Failed tests:

test.test_asyncio.test_subprocess

Failed subtests:

test_subprocess_consistent_callbacks - test.test_asyncio.test_subprocess.SubprocessThreadedWatcherTests.test_subprocess_consistent_callbacks

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/Lib/test/test_asyncio/test_subprocess.py", line 788, in test_subprocess_consistent_callbacks
    self.loop.run_until_complete(main())
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/Lib/asyncio/base_events.py", line 664, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/Lib/test/test_asyncio/test_subprocess.py", line 780, in main
    self.assertEqual(events, [
AssertionError: Lists differ: [('pi[29 chars]t'), 'pipe_connection_lost', ('pipe_data_recei[57 chars]ted'] != [('pi[29 chars]t'), ('pipe_data_received', 2, b'stderr'), 'pi[57 chars]ted']

bedevere-bot · 2023-09-17T12:03:31Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian root 3.x has failed when building commit add16f1.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/345/builds/5854) and take a look at the build logs.
Check if the failure is related to this commit (add16f1) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/345/builds/5854

Failed tests:

test_signal

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_signal.py", line 1287, in test_stress_delivery_dependent
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 4543 != 10000 : Some signals were lost

bedevere-bot · 2023-09-17T12:17:36Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot PPC64LE RHEL7 LTO + PGO 3.x has failed when building commit add16f1.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/43/builds/4241) and take a look at the build logs.
Check if the failure is related to this commit (add16f1) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/43/builds/4241

Failed tests:

test.test_asyncio.test_unix_events
test.test_multiprocessing_fork.test_misc

Failed subtests:

test_fork_signal_handling - test.test_asyncio.test_unix_events.TestFork.test_fork_signal_handling
test_nested_startmethod - test.test_multiprocessing_fork.test_misc.TestStartMethod.test_nested_startmethod

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/_test_multiprocessing.py", line 5475, in test_nested_startmethod
    self.assertEqual(results, [2, 1])
AssertionError: Lists differ: [1, 2] != [2, 1]


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/unittest/async_case.py", line 90, in _callTestMethod
    if self._callMaybeAsync(method) is not None:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/unittest/async_case.py", line 117, in _callMaybeAsync
    return self._asyncioTestContext.run(func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/support/hashlib_helper.py", line 49, in wrapper
    return func_or_class(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_asyncio/test_unix_events.py", line 1937, in test_fork_signal_handling
    self.assertTrue(child_handled.is_set())
AssertionError: False is not true

bedevere-bot · 2023-09-17T12:37:10Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot PPC64LE RHEL8 LTO 3.x has failed when building commit add16f1.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/361/builds/4062) and take a look at the build logs.
Check if the failure is related to this commit (add16f1) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/361/builds/4062

Failed tests:

test.test_asyncio.test_unix_events
test.test_asyncio.test_subprocess

Failed subtests:

test_subprocess_consistent_callbacks - test.test_asyncio.test_subprocess.SubprocessThreadedWatcherTests.test_subprocess_consistent_callbacks
test_fork_signal_handling - test.test_asyncio.test_unix_events.TestFork.test_fork_signal_handling

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto/build/Lib/test/test_asyncio/test_subprocess.py", line 788, in test_subprocess_consistent_callbacks
    self.loop.run_until_complete(main())
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto/build/Lib/asyncio/base_events.py", line 664, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto/build/Lib/test/test_asyncio/test_subprocess.py", line 780, in main
    self.assertEqual(events, [
AssertionError: Lists differ: ['process_exited', ('pipe_data_received', 1, b'stdout')] != [('pipe_data_received', 1, b'stdout'), ('p[95 chars]ted']


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto/build/Lib/unittest/async_case.py", line 90, in _callTestMethod
    if self._callMaybeAsync(method) is not None:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto/build/Lib/unittest/async_case.py", line 117, in _callMaybeAsync
    return self._asyncioTestContext.run(func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto/build/Lib/test/support/hashlib_helper.py", line 49, in wrapper
    return func_or_class(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto/build/Lib/test/test_asyncio/test_unix_events.py", line 1937, in test_fork_signal_handling
    self.assertTrue(child_handled.is_set())
AssertionError: False is not true

bedevere-bot · 2023-09-17T12:50:22Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot PPC64LE RHEL7 LTO 3.x has failed when building commit add16f1.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/503/builds/3918) and take a look at the build logs.
Check if the failure is related to this commit (add16f1) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/503/builds/3918

Failed tests:

test.test_asyncio.test_unix_events

Failed subtests:

test_fork_signal_handling - test.test_asyncio.test_unix_events.TestFork.test_fork_signal_handling

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-ppc64le.lto/build/Lib/unittest/async_case.py", line 90, in _callTestMethod
    if self._callMaybeAsync(method) is not None:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-ppc64le.lto/build/Lib/unittest/async_case.py", line 117, in _callMaybeAsync
    return self._asyncioTestContext.run(func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-ppc64le.lto/build/Lib/test/support/hashlib_helper.py", line 49, in wrapper
    return func_or_class(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-ppc64le.lto/build/Lib/test/test_asyncio/test_unix_events.py", line 1937, in test_fork_signal_handling
    self.assertTrue(child_handled.is_set())
AssertionError: False is not true

vstinner · 2023-09-18T17:52:08Z

I wrote a first PR to add PyMapping_HasKeyWithError() and PyMapping_HasKeyStringWithError() functions to pythoncapi-compat: python/pythoncapi-compat#73

…ors (pythonGH-109025) Add the following functions: * PyObject_HasAttrWithError() * PyObject_HasAttrStringWithError() * PyMapping_HasKeyWithError() * PyMapping_HasKeyStringWithError()

scoder · 2023-10-15T10:45:44Z

Doc/c-api/mapping.rst

+
+   Return ``1`` if the mapping object has the key *key* and ``0`` otherwise.
+   This is equivalent to the Python expression ``key in o``.
+   On failure, return ``-1``.


Given the relevance of error handling for these new functions, I think the docs should mention explicitly under which conditions users must expect exceptions.

Suggested change

On failure, return ``-1``.

On failure, return ``-1`` with an exception set.

Sometimes I write "On error, raise an exception and return -1." Even if in C, technically, it's more "setting" an exception, I like to use Python "raise" semantics in C, it's easy for me to map C/Python this way :-)

scoder · 2023-10-15T10:46:02Z

Doc/c-api/object.rst

+
+   Returns ``1`` if *o* has the attribute *attr_name*, and ``0`` otherwise.
+   This is equivalent to the Python expression ``hasattr(o, attr_name)``.
+   On failure, return ``-1``.


Given the relevance of error handling for these new functions, I think the docs should mention explicitly under which conditions users must expect exceptions.

Suggested change

On failure, return ``-1``.

On failure, return ``-1`` with an exception set.

The simulator needs to be tested with Python 3.12 - it presently does not fully work on Python 3.13 due to the change in behaviour for `PyObject_HasAttr()` to no longer silently ignore all errors as it did previously. Numba's typecode fingerprinting calls something which is approximately `PyObject_HasAttr("_numba_type_")` on NumPy arrays during some execution paths of the simulator, which causes NumPy to raise `ValueError("no field of name _numba_type_")`. Since Python 3.13 this causes masses of warnings to be raised that flood the output. References: - Python 3.12 `PyObject_HasAttr()`: https://docs.python.org/3.12/c-api/object.html#c.PyObject_HasAttr > "Exceptions that occur when this calls __getattr__() and > __getattribute__() methods are silently ignored." - Python 3.13 `PyObject_HasAttr()`: https://docs.python.org/3.13/c-api/object.html#c.PyObject_HasAttr > "Exceptions that occur when this calls __getattr__() and > __getattribute__() methods aren’t propagated, but instead given to > sys.unraisablehook(). For proper error handling, use > PyObject_HasAttrWithError(), ..." - CPython commit adding the recommended functions: python/cpython#109025 - CPython issue: python/cpython#108511

pythongh-108511: Add C API functions which do not silently ignore errors

ae6e789

Add the following functions: * PyObject_HasAttrWithError() * PyObject_HasAttrStringWithError() * PyMapping_HasKeyWithError() * PyMapping_HasKeyStringWithError()

serhiy-storchaka added type-feature A feature request or enhancement topic-C-API labels Sep 6, 2023

serhiy-storchaka requested review from kumaraditya303, markshannon and methane as code owners September 6, 2023 20:13

bedevere-bot added the awaiting core review label Sep 6, 2023

bedevere-bot mentioned this pull request Sep 6, 2023

C API: Add replacements for PyObject_HasAttr() etc #108511

Closed

regen-limited-abi

7bcdfad

serhiy-storchaka requested review from a team and encukou as code owners September 6, 2023 20:45

kumaraditya303 removed their request for review September 12, 2023 17:46

methane approved these changes Sep 12, 2023

View reviewed changes

serhiy-storchaka merged commit add16f1 into python:main Sep 17, 2023

bedevere-app bot removed the awaiting core review label Sep 17, 2023

serhiy-storchaka deleted the PyObject_HasAttrWithError-PyMapping_HasKeyWithError branch September 17, 2023 11:23

scoder reviewed Oct 15, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

gh-108511: Add C API functions which do not silently ignore errors #109025

gh-108511: Add C API functions which do not silently ignore errors #109025

Uh oh!

serhiy-storchaka commented Sep 6, 2023 •

edited by github-actions bot

Loading

Uh oh!

bedevere-bot commented Sep 17, 2023

Uh oh!

bedevere-bot commented Sep 17, 2023

Uh oh!

bedevere-bot commented Sep 17, 2023

Uh oh!

bedevere-bot commented Sep 17, 2023

Uh oh!

bedevere-bot commented Sep 17, 2023

Uh oh!

bedevere-bot commented Sep 17, 2023

Uh oh!

bedevere-bot commented Sep 17, 2023

Uh oh!

vstinner commented Sep 18, 2023

Uh oh!

scoder Oct 15, 2023

Uh oh!

vstinner Oct 16, 2023

Uh oh!

scoder Oct 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	On failure, return ``-1``.
	On failure, return ``-1`` with an exception set.

Uh oh!

gh-108511: Add C API functions which do not silently ignore errors #109025

gh-108511: Add C API functions which do not silently ignore errors #109025

Uh oh!

Conversation

serhiy-storchaka commented Sep 6, 2023 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-bot commented Sep 17, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Sep 17, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Sep 17, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Sep 17, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Sep 17, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Sep 17, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Sep 17, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

vstinner commented Sep 18, 2023

Uh oh!

scoder Oct 15, 2023

Choose a reason for hiding this comment

Uh oh!

vstinner Oct 16, 2023

Choose a reason for hiding this comment

Uh oh!

scoder Oct 15, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

serhiy-storchaka commented Sep 6, 2023 •

edited by github-actions bot

Loading