From 1b10658f7640abe330f9e8bf5673a0a29e7ecdc8 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Tue, 11 Mar 2025 12:46:15 +0000 Subject: [PATCH 1/4] Document some examples and notes about non-Python threads with subinterpreters. --- Doc/c-api/init.rst | 54 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 48 insertions(+), 6 deletions(-) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 9197f704fab344..dfe7a271d3fd03 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1064,8 +1064,37 @@ Note that the ``PyGILState_*`` functions assume there is only one global interpreter (created automatically by :c:func:`Py_Initialize`). Python supports the creation of additional interpreters (using :c:func:`Py_NewInterpreter`), but mixing multiple interpreters and the -``PyGILState_*`` API is unsupported. +``PyGILState_*`` API is unsupported. This is because :c:func:`PyGILState_Ensure` +and similar functions almost always acquire the :term:`GIL` of the main interpreter +in threads created by subinterpreters, which can lead to data races or deadlocks. + +Supporting subinterpreters in non-Python threads +------------------------------------------------ + +If you would like to support subinterpreters with non-Python created threads, you +must use the ``PyThreadState_*`` API instead of the traditional ``PyGILState_*`` +API. + +In particular, you must store the exact interpreter state from the calling +function and pass it to :c:func:`PyThreadState_New` to ensure that the thread +acquires the :term:`GIL` of the subinterpreter instead of the main interpreter. +In turn, this means that the return value of :c:func:`PyInterpreterState_Get` +should be stored alongside any information passed to the thread.:: + + /* The return value of PyInterpreterState_Get() from the + function that created this thread. */ + PyInterpreterState *interp = ThreadData->interp; + PyThreadState *tstate = PyThreadState_New(interp); + PyThreadState_Swap(tstate); + + /* GIL of the subinterpreter is now held. + Perform Python actions here. */ + result = CallSomeFunction(); + /* evaluate result or handle exception */ + /* Destroy the thread state. No Python API allowed beyond this point. */ + PyThreadState_Clear(tstate); + PyThreadState_DeleteCurrent(); .. _fork-and-threads: @@ -1243,6 +1272,10 @@ code, or when embedding the Python interpreter: *tstate*, which may be ``NULL``. The global interpreter lock must be held and is not released. + .. note:: + Similar to :c:func:`PyGILState_Ensure`, this function will hang the + thread if the runtime is finalizing. + The following functions use thread-local storage, and are not compatible with sub-interpreters: @@ -1269,10 +1302,10 @@ with sub-interpreters: When the function returns, the current thread will hold the GIL and be able to call arbitrary Python code. Failure is a fatal error. - .. note:: - Calling this function from a thread when the runtime is finalizing will - hang the thread until the program exits, even if the thread was not - created by Python. Refer to + .. warning:: + Calling this function when the runtime is finalizing is unsafe. Doing + so will either hang the thread until the program ends, or fully crash + the interpreter in rare cases. Refer to :ref:`cautions-regarding-runtime-finalization` for more details. .. versionchanged:: 3.14 @@ -1289,7 +1322,6 @@ with sub-interpreters: Every call to :c:func:`PyGILState_Ensure` must be matched by a call to :c:func:`PyGILState_Release` on the same thread. - .. c:function:: PyThreadState* PyGILState_GetThisThreadState() Get the current thread state for this thread. May return ``NULL`` if no @@ -1297,6 +1329,11 @@ with sub-interpreters: always has such a thread-state, even if no auto-thread-state call has been made on the main thread. This is mainly a helper/diagnostic function. + .. note:: + This function does not account for thread states created by something + other than :c:func:`PyGILState_Ensure` (such as :c:func:`PyThreadState_New`). + Prefer :c:func:`PyThreadState_Get` or :c:func:`PyThreadState_GetUnchecked` + for most cases. .. c:function:: int PyGILState_Check() @@ -1309,6 +1346,11 @@ with sub-interpreters: knowing that the GIL is locked can allow the caller to perform sensitive actions or otherwise behave differently. + .. note:: + If the current Python process has ever created a subinterpreter, this + function will *always* return ``1``. Prefer :c:func:`PyThreadState_GetUnchecked` + for most cases. + .. versionadded:: 3.4 From 1afcbb7139b2640711b9f054950e517759bd78f6 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Wed, 19 Mar 2025 12:22:30 +0000 Subject: [PATCH 2/4] Add clarification to 'only if' --- Doc/c-api/init.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index dfe7a271d3fd03..9d101f33e3ab36 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1339,8 +1339,8 @@ with sub-interpreters: Return ``1`` if the current thread is holding the GIL and ``0`` otherwise. This function can be called from any thread at any time. - Only if it has had its Python thread state initialized and currently is - holding the GIL will it return ``1``. + Only if it has had its Python thread state initialized via :c:func:`PyGILState_Ensure` + and currently is holding the GIL will it return ``1``. This is mainly a helper/diagnostic function. It can be useful for example in callback contexts or memory allocation functions when knowing that the GIL is locked can allow the caller to perform sensitive From 660aa77bfa27d16ff4bd8ce9c35f1ccc0c88f01d Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 24 Mar 2025 10:43:26 -0400 Subject: [PATCH 3/4] Switch to thread state terms instead of the silly GIL --- Doc/c-api/init.rst | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index cc2962e9ad3544..f2e85a4a866d68 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1084,8 +1084,9 @@ interpreter (created automatically by :c:func:`Py_Initialize`). Python supports the creation of additional interpreters (using :c:func:`Py_NewInterpreter`), but mixing multiple interpreters and the ``PyGILState_*`` API is unsupported. This is because :c:func:`PyGILState_Ensure` -and similar functions almost always acquire the :term:`GIL` of the main interpreter -in threads created by subinterpreters, which can lead to data races or deadlocks. +and similar functions default to :term:`attaching ` a +:term:`thread state` for the main interpreter, meaning that the thread can't safely +interact with the calling subinterpreter. Supporting subinterpreters in non-Python threads ------------------------------------------------ @@ -1094,11 +1095,9 @@ If you would like to support subinterpreters with non-Python created threads, yo must use the ``PyThreadState_*`` API instead of the traditional ``PyGILState_*`` API. -In particular, you must store the exact interpreter state from the calling -function and pass it to :c:func:`PyThreadState_New` to ensure that the thread -acquires the :term:`GIL` of the subinterpreter instead of the main interpreter. -In turn, this means that the return value of :c:func:`PyInterpreterState_Get` -should be stored alongside any information passed to the thread.:: +In particular, you must store the interpreter state from the calling +function and pass it to :c:func:`PyThreadState_New`, which will ensure that +the :term:`thread state` is targeting the correct interpreter.:: /* The return value of PyInterpreterState_Get() from the function that created this thread. */ From 961d8b90b301044e7b945f645d7760dca1e6b499 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 5 May 2025 07:43:07 -0400 Subject: [PATCH 4/4] Update init.rst Co-authored-by: Victor Stinner --- Doc/c-api/init.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index f2e85a4a866d68..12484fb58d4701 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1097,7 +1097,7 @@ API. In particular, you must store the interpreter state from the calling function and pass it to :c:func:`PyThreadState_New`, which will ensure that -the :term:`thread state` is targeting the correct interpreter.:: +the :term:`thread state` is targeting the correct interpreter:: /* The return value of PyInterpreterState_Get() from the function that created this thread. */