bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread #14344

asvetlov · 2019-06-24T15:30:15Z

The resurrection of #13630
Now without dangling threads in tests :)

https://bugs.python.org/issue35621

…en loop is executed in non-main thread (python#13630)" (pythonGH-13793)" This reverts commit 9535aff.

asvetlov · 2019-06-24T15:38:22Z

I tested it with manual adding 1-second timeout in spawned waiter threads to emulate slow test machine.
After making sure that ./python -Werror -m test -v --fail-env-changed test_asyncio doesn't produce an error I removed the sleep.

asvetlov · 2019-06-24T15:43:13Z

Lib/asyncio/unix_events.py

@@ -1035,11 +1052,6 @@ def __exit__(self, a, b, c):
    def add_child_handler(self, pid, callback, *args):
        assert self._forks, "Must use the context manager"

-        if self._loop is None:


The check is replaced by watcher.is_active() call in subprocess transport

1st1

LGTM overall. Thanks for pushing this through, Andrew.

Doc/library/asyncio-policy.rst

Doc/library/asyncio-subprocess.rst

Lib/asyncio/unix_events.py

Co-Authored-By: Yury Selivanov <[email protected]>

asvetlov · 2019-06-30T09:00:21Z

Thanks for review!

miss-islington · 2019-06-30T09:55:02Z

Thanks @asvetlov for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

bedevere-bot · 2019-06-30T09:55:16Z

GH-14484 is a backport of this pull request to the 3.8 branch.

…ted in non-main thread (pythonGH-14344) (cherry picked from commit 0d671c0) Co-authored-by: Andrew Svetlov <[email protected]>

…ted in non-main thread (GH-14344) (cherry picked from commit 0d671c0) Co-authored-by: Andrew Svetlov <[email protected]>

…ted in non-main thread (pythonGH-14344)

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

PR #5862 Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344 * https://docs.python.org/3/library/asyncio-policy.html#asyncio.MultiLoopChildWatcher Co-authored-by: Sviatoslav Sydorenko <[email protected]>

PR #5862 Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344 * https://docs.python.org/3/library/asyncio-policy.html#asyncio.MultiLoopChildWatcher Co-authored-by: Sviatoslav Sydorenko <[email protected]> (cherry picked from commit 7080a8b)

With Python 3.7 asynchronous tests randomly fail with "RuntimeError: Event loop is closed" during shutdown. Backtrace of the error includes `SafeChildWatcher` calls. Python 3.8 has replaced `SafeChildWatcher` with a new `ThreadedChildWatcher` by default [1] as a bugfix for "asyncio.create_subprocess_exec() only works with main event loop" bug [2]. Python 3.7 scheduled end of life is 2023-06-27 according to <https://devguide.python.org/versions/>. [1] python/cpython#14344 [2] https://bugs.python.org/issue35621

asvetlov added 7 commits June 4, 2019 02:32

Close pid waiters on watcher.close()

3695efd

Revert "Revert "bpo-35621: Support running subprocesses in asyncio wh…

3559581

…en loop is executed in non-main thread (python#13630)" (pythonGH-13793)" This reverts commit 9535aff.

Merge branch 'dangling-threads-in-watcher' into threaded-waiter

b18e6b9

Merge remote-tracking branch 'upstream/master' into threaded-waiter

1bc1d8e

Work on

ef1e21c

Merge branch 'master' into threaded-waiter

c982e4a

Fix dangling thread warnings in tests

6912ec7

asvetlov requested review from 1st1 and gpshead as code owners June 24, 2019 15:30

the-knights-who-say-ni added the CLA signed label Jun 24, 2019

bedevere-bot added the awaiting core review label Jun 24, 2019

asvetlov commented Jun 24, 2019

View reviewed changes

Update documentation

efb19c0

asvetlov added topic-asyncio needs backport to 3.8 labels Jun 24, 2019

Add docstrings

eb07403

1st1 approved these changes Jun 30, 2019

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Jun 30, 2019

asvetlov and others added 8 commits June 30, 2019 11:50

Update Doc/library/asyncio-policy.rst

6aa3357

Co-Authored-By: Yury Selivanov <[email protected]>

Fix doc text

25d9ac2

Update Doc/library/asyncio-policy.rst

ae3a706

Co-Authored-By: Yury Selivanov <[email protected]>

Update Doc/library/asyncio-policy.rst

8baf48c

Co-Authored-By: Yury Selivanov <[email protected]>

Update Doc/library/asyncio-policy.rst

6e6a783

Co-Authored-By: Yury Selivanov <[email protected]>

Update Lib/asyncio/unix_events.py

1e1281a

Co-Authored-By: Yury Selivanov <[email protected]>

Update Lib/asyncio/unix_events.py

936aeb0

Co-Authored-By: Yury Selivanov <[email protected]>

Update docs

cc5a752

asvetlov merged commit 0d671c0 into python:master Jun 30, 2019

bedevere-bot removed the awaiting merge label Jun 30, 2019

asvetlov deleted the threaded-waiter branch June 30, 2019 09:55

bedevere-bot removed the needs backport to 3.8 label Jun 30, 2019

miss-islington added a commit that referenced this pull request Jun 30, 2019

bpo-35621: Support running subprocesses in asyncio when loop is execu…

bf8cb31

…ted in non-main thread (GH-14344) (cherry picked from commit 0d671c0) Co-authored-by: Andrew Svetlov <[email protected]>

lisroach pushed a commit to lisroach/cpython that referenced this pull request Sep 10, 2019

bpo-35621: Support running subprocesses in asyncio when loop is execu…

af4d7c9

…ted in non-main thread (pythonGH-14344)

DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020

bpo-35621: Support running subprocesses in asyncio when loop is execu…

b340eb2

…ted in non-main thread (pythonGH-14344)

webknjaz mentioned this pull request Jan 25, 2021

pytest-xdist breaks asyncio-based code expecting to be run in the main thread pytest-dev/pytest-xdist#620

Closed

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Jan 25, 2021

Use ThreadedChildWatcher in tests where available

a7f5a2a

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Jan 25, 2021

Use ThreadedChildWatcher in tests where available

7b7a324

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

webknjaz mentioned this pull request Jan 25, 2021

Re-introduce pytest-xdist in supported envs aio-libs/aiohttp#5431

Merged

5 tasks

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Jan 25, 2021

Use ThreadedChildWatcher in tests where available

e3df8bc

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Feb 21, 2021

Use ThreadedChildWatcher in tests where available

0df32d5

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Feb 24, 2021

Use ThreadedChildWatcher in tests where available

bea959a

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

link2xt mentioned this pull request May 19, 2023

"Event loop is closed" error in _provide_clean_event_loop() pytest-dev/pytest-asyncio#548

Closed

link2xt mentioned this pull request May 19, 2023

ci: require Python 3.8 for deltachat-rpc-client chatmail/core#4410

Merged

gvanrossum mentioned this pull request Oct 15, 2023

test_asyncio.test_subprocess leaked dangling threads on PPC64LE Fedora Stable 3.x: #110205

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread #14344

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread #14344

asvetlov commented Jun 24, 2019 •

edited by bedevere-bot

Loading

asvetlov commented Jun 24, 2019

asvetlov Jun 24, 2019

1st1 left a comment

asvetlov commented Jun 30, 2019

miss-islington commented Jun 30, 2019

bedevere-bot commented Jun 30, 2019

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread #14344

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread #14344

Conversation

asvetlov commented Jun 24, 2019 • edited by bedevere-bot Loading

asvetlov commented Jun 24, 2019

asvetlov Jun 24, 2019

Choose a reason for hiding this comment

1st1 left a comment

Choose a reason for hiding this comment

asvetlov commented Jun 30, 2019

miss-islington commented Jun 30, 2019

bedevere-bot commented Jun 30, 2019

asvetlov commented Jun 24, 2019 •

edited by bedevere-bot

Loading