PyGILState_Ensure on non-Python thread causes fatal error

[zooba]

BPO	20891
Nosy	@gvanrossum, @warsaw, @ncoghlan, @pitrou, @vstinner, @ericsnowcurrently, @zooba
PRs	bpo-20891: Fix PyGILState_Ensure() #4650 [3.6] bpo-20891: Fix PyGILState_Ensure() (#4650) #4655 [2.7] bpo-20891: Fix PyGILState_Ensure() (#4650) #4657 bpo-20891: Py_Initialize() now creates the GIL #4700 bpo-20891: Skip test_embed.test_bpo20891() #4967 [3.6] bpo-20891: Skip test_embed.test_bpo20891() (#4967) #4969 bpo-20891: Reenable test_embed.test_bpo20891() #5420 [3.6] bpo-20891: Py_Initialize() now creates the GIL (#4700) #5421 [3.6] bpo-20891: Remove test_capi.test_bpo20891() #5425
Files	test.c ptest.c PyGILState_Ensure.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-01-29.13:07:27.074>
created_at = <Date 2014-03-11.18:40:13.148>
labels = ['extension-modules', '3.7']
title = 'PyGILState_Ensure on non-Python thread causes fatal error'
updated_at = <Date 2018-01-29.15:35:52.778>
user = 'https://github.com/zooba'

bugs.python.org fields:

activity = <Date 2018-01-29.15:35:52.778>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-01-29.13:07:27.074>
closer = 'vstinner'
components = ['Extension Modules']
creation = <Date 2014-03-11.18:40:13.148>
creator = 'steve.dower'
dependencies = []
files = ['34358', '42173', '42174']
hgrepos = []
issue_num = 20891
keywords = ['patch']
message_count = 31.0
messages = ['213160', '213161', '213398', '261820', '261821', '307038', '307330', '307331', '307342', '307349', '307350', '307351', '307550', '307551', '307552', '307553', '307554', '307571', '307583', '307586', '307628', '308588', '308914', '308917', '311090', '311091', '311095', '311099', '311124', '311125', '311143']
nosy_count = 10.0
nosy_names = ['gvanrossum', 'barry', 'ncoghlan', 'pitrou', 'vstinner', 'Mekk', 'kchen', 'eric.snow', 'steve.dower', 'xcombelle']
pr_nums = ['4650', '4655', '4657', '4700', '4967', '4969', '5420', '5421', '5425']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue20891'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7']

[zooba]

In Python 3.4rc3, calling PyGILState_Ensure() from a thread that was not created by Python and without any calls to PyEval_InitThreads() will cause a fatal exit:

Fatal Python error: take_gil: NULL tstate

I believe this was added in http://hg.python.org/cpython/rev/dc4e805ec68a. The attached file has a minimal repro (for Windows, sorry, but someone familiar with low-level *nix threading APIs should be able to translate it easily).

Basically, the sequence looks like this:

1. main() calls Py_Initialize()
2. main() starts new C-level thread
3. main() calls Py_BEGIN_ALLOW_THREADS and waits for the new thread
4. New thread calls PyGILState_Ensure()
5. PyGILState_Ensure() sees that there is no thread state and calls PyEval_InitThreads() (this is what changed in the linked changeset)
6. PyEval_InitThreads() creates the GIL
7. PyEval_InitThreads() calls take_gil(PyThreadState_GET()) (this is what aborts - there is still no thread state)
8. (expected) PyGILState_Ensure() calls PyThreadState_New() and PyEval_RestoreThread().

The attached repro has two workarounds (commented out). The first is to call PyThreadState_New()/PyEval_RestoreThread() from the new thread, and the second is to call PyEval_InitThreads() from the main thread.

The latter is inappropriate for my use, since I am writing a debugger that can inject itself into a running CPython process - there is no way to require that PyEval_InitThreads() has been called and I can't reliably inject it onto the main thread. The first workaround is okay, but I'd rather not have to special-case 3.4 by rewriting PyGILState_Ensure() so that I can safely call PyGILState_Ensure().

[zooba]

Should have linked to bpo-19576 as well, which is the issue associated with that changeset.

[vstinner]

IMO it's a bug in PyEval_InitThreads().

[vstinner]

ptest.c: Portable example using Python API.

[vstinner]

Attached patch should fix the issue.

[Mekk]

Is this fix released? I can't find it in the changelog…

(I faced this bug on 3.5.2, released a couple of months after this bug was closed…)

[vstinner]

Is this fix released? I can't find it in the changelog…

Oops, I lost track of this issue, but it wasn't fixed, no.

I just proposed my old fix as a pull requets: PR 4650.

(I faced this bug on 3.5.2, released a couple of months after this bug was closed…)

The workaround is to call PyEval_InitThreads() before spawning your first non-Python thread.
https://docs.python.org/dev/c-api/init.html#c.PyEval_InitThreads

[vstinner]

I have to check if Python 2.7 is impacted as well.

[vstinner]

New changeset b4d1e1f by Victor Stinner in branch 'master':
bpo-20891: Fix PyGILState_Ensure() (bpo-4650)
b4d1e1f

[vstinner]

New changeset be6b74c by Victor Stinner in branch '2.7':
bpo-20891: Fix PyGILState_Ensure() (bpo-4650) (bpo-4657)
be6b74c

[vstinner]

New changeset e10c9de by Victor Stinner in branch '3.6':
bpo-20891: Fix PyGILState_Ensure() (bpo-4650) (bpo-4655)
e10c9de

[vstinner]

Ok, the bug is now fixed in Python 2.7, 3.6 and master (future 3.7). On 3.6 and master, the fix comes with an unit test.

Thanks Steve Dower for the bug report, sorry for the long delay, I completely forgot this old bug!

Marcin Kasperski: Thanks for the reminder. Sadly, we will have to wait for the next release to get the fix. In the meanwhile, you can workaround the bug by calling PyEval_InitThreads() after Py_Initialize() but before running actual code.