Skip to content

warnings.catch_warnings is not thread safe or async safe #128384

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
nascheme opened this issue Dec 31, 2024 · 1 comment
Closed

warnings.catch_warnings is not thread safe or async safe #128384

nascheme opened this issue Dec 31, 2024 · 1 comment
Labels
stdlib Python modules in the Lib dir topic-asyncio topic-free-threading type-bug An unexpected behavior, bug, or error

Comments

@nascheme
Copy link
Member

nascheme commented Dec 31, 2024

Bug report

Bug description:

This is an old issue but it has become more urgent to fix given free-threaded Python. A simple example to show how the current context manager can fail:

from concurrent.futures import ThreadPoolExecutor

import warnings

def test_warning():
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        warnings.warn("my warning", UserWarning)

tpe = ThreadPoolExecutor(max_workers=10)

threads = [tpe.submit(test_warning) for _ in range(100)]
[t.result() for t in threads]

Since test_warning() is running in separate threads, there is no consistent ordering of which context manager exits first. That means the filters value that gets restored is also not consistent (a race between the threads about which version of the filters value they will restore).

The context manager is also not friendly to async code. These issues have been reported before:

The fairly obvious fix to this issue is to use contextvars, similar to how the decimal module uses a context. This is both thread safe and friendly to async code.

The warnings module itself has some "shallow" thread safety issues but these are relatively easy to fix and no API changes are needed.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

@nascheme nascheme added type-bug An unexpected behavior, bug, or error stdlib Python modules in the Lib dir topic-free-threading labels Dec 31, 2024
@github-project-automation github-project-automation bot moved this to Todo in asyncio Jan 1, 2025
nascheme added a commit that referenced this issue Jan 14, 2025
nascheme added a commit to nascheme/cpython that referenced this issue Feb 12, 2025
nascheme added a commit to nascheme/cpython that referenced this issue Feb 21, 2025
nascheme added a commit to nascheme/cpython that referenced this issue Mar 11, 2025
nascheme added a commit to nascheme/cpython that referenced this issue Mar 27, 2025
nascheme added a commit to nascheme/cpython that referenced this issue Mar 27, 2025
nascheme added a commit to nascheme/cpython that referenced this issue Mar 27, 2025
nascheme added a commit to nascheme/cpython that referenced this issue Apr 9, 2025
nascheme added a commit that referenced this issue Apr 9, 2025
)

Make `warnings.catch_warnings()` use a context variable for holding
the warning filtering state if the `sys.flags.context_aware_warnings`
flag is set to true.  This makes using the context manager thread-safe in
multi-threaded programs.

Add the `sys.flags.thread_inherit_context` flag.  If true, starting a new
thread with `threading.Thread` will use a copy of the context
from the caller of `Thread.start()`.

Both these flags are set to true by default for the free-threaded build
and false for the default build.

Move the Python implementation of warnings.py into _py_warnings.py.

Make _contextvars a builtin module.

Co-authored-by: Kumar Aditya <[email protected]>
@nascheme
Copy link
Member Author

Fixed by GH-130010. You have to set the sys.flags.context_aware_warnings flag to ensure you get the new behavior. If you are using threads, you will want to set sys.flags.thread_inherit_context as well.

@github-project-automation github-project-automation bot moved this from Todo to Done in asyncio Apr 14, 2025
seehwan pushed a commit to seehwan/cpython that referenced this issue Apr 16, 2025
…ythongh-130010)

Make `warnings.catch_warnings()` use a context variable for holding
the warning filtering state if the `sys.flags.context_aware_warnings`
flag is set to true.  This makes using the context manager thread-safe in
multi-threaded programs.

Add the `sys.flags.thread_inherit_context` flag.  If true, starting a new
thread with `threading.Thread` will use a copy of the context
from the caller of `Thread.start()`.

Both these flags are set to true by default for the free-threaded build
and false for the default build.

Move the Python implementation of warnings.py into _py_warnings.py.

Make _contextvars a builtin module.

Co-authored-by: Kumar Aditya <[email protected]>
hugovk added a commit to hugovk/cpython that referenced this issue Apr 16, 2025
nascheme added a commit to nascheme/cpython that referenced this issue Apr 16, 2025
nascheme added a commit that referenced this issue Apr 16, 2025
When the `showwarning()` function is replaced, make sure to restore
it after the test finishes.  Add a timeout for `Barrier()` so we
don't hang for a long time if something goes wrong.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stdlib Python modules in the Lib dir topic-asyncio topic-free-threading type-bug An unexpected behavior, bug, or error
Projects
Status: Done
Development

No branches or pull requests

2 participants