Replace context enter and exit events with single "context switched" event

[rhansen]

Feature or enhancement

Proposal:

(split off of #99633 (comment) to keep related pull requests and discussion better organized)

Starting in v3.14, C code can subscribe to "context enter" and "context exit" events:

cpython/Include/cpython/context.h

Lines 30 to 60 in 120729d

	typedef enum {
	Py_CONTEXT_EVENT_ENTER,
	Py_CONTEXT_EVENT_EXIT,
	} PyContextEvent;
	/*
	* Callback to be invoked when a context object is entered or exited.
	*
	* The callback is invoked with the event and a reference to
	* the context after its entered and before its exited.
	*
	* if the callback returns with an exception set, it must return -1. Otherwise
	* it should return 0
	*/
	typedef int (*PyContext_WatchCallback)(PyContextEvent, PyContext *);
	/*
	* Register a per-interpreter callback that will be invoked for context object
	* enter/exit events.
	*
	* Returns a handle that may be passed to PyContext_ClearWatcher on success,
	* or -1 and sets and error if no more handles are available.
	*/
	PyAPI_FUNC(int) PyContext_AddWatcher(PyContext_WatchCallback callback);
	/*
	* Clear the watcher associated with the watcher_id handle.
	*
	* Returns 0 on success or -1 if no watcher exists for the provided id.
	*/
	PyAPI_FUNC(int) PyContext_ClearWatcher(int watcher_id);

See gh-119333 for some background.

I would like to replace the just-added "context enter" and "context exit" events with a single "context switched" event:

diff --git a/Include/cpython/context.h b/Include/cpython/context.h
index ec72966e82c..f9638b0abe5 100644
--- a/Include/cpython/context.h
+++ b/Include/cpython/context.h
@@ -28,8 +28,12 @@ PyAPI_FUNC(int) PyContext_Enter(PyObject *);
 PyAPI_FUNC(int) PyContext_Exit(PyObject *);
 
 typedef enum {
-   Py_CONTEXT_EVENT_ENTER,
-   Py_CONTEXT_EVENT_EXIT,
+    /*
+     * The current context has switched to a different context.  The object
+     * passed to the watch callback is the now-current contextvars.Context
+     * object.
+     */
+    Py_CONTEXT_SWITCHED = 1,
 } PyContextEvent;
 
 /*
diff --git a/Python/context.c b/Python/context.c
index ddb03555f9e..6c9f68583c2 100644
--- a/Python/context.c
+++ b/Python/context.c
@@ -192,7 +192,7 @@ _PyContext_Enter(PyThreadState *ts, PyObject *octx)
     ts->context = Py_NewRef(ctx);
     ts->context_ver++;
 
-    notify_context_watchers(Py_CONTEXT_EVENT_ENTER, ctx, ts);
+    notify_context_watchers(Py_CONTEXT_SWITCHED, ctx, ts);
     return 0;
 }
 
@@ -226,13 +226,13 @@ _PyContext_Exit(PyThreadState *ts, PyObject *octx)
         return -1;
     }
 
-    notify_context_watchers(Py_CONTEXT_EVENT_EXIT, ctx, ts);
     Py_SETREF(ts->context, (PyObject *)ctx->ctx_prev);
     ts->context_ver++;
 
     ctx->ctx_prev = NULL;
     ctx->ctx_entered = 0;
 
+    notify_context_watchers(Py_CONTEXT_SWITCHED, (PyContext *)ts->context, ts);
     return 0;
 }
 

Rationale: Users want to know when the current context switches to a different context object. Right now this happens when and only when a context is entered or exited, so the enter and exit events are synonymous with "switched". However, if the changes proposed for gh-99633 are implemented, the current context will also switch for reasons other than context enter or exit. Since users actually care about context switches and not enter or exit, replacing the enter and exit events with a single switched event would make the new feature compatible with gh-99633.

The current "context exit" event is emitted just before exiting the context. The new "context switched" event would be emitted after the context is exited to match the semantics users expect of an event with a past-tense name. If users need the ability to clean up before the switch takes effect, another "context is about to switch" event can be added in the future. I am not proposing to add such an event for this issue because no users have requested it yet (YAGNI).

I would love to get this resolved before v3.14 is released so that we don't have to worry about breaking backwards compatibility.

Has this already been discussed elsewhere?

I have already discussed this feature proposal on another bug report.

Links to previous discussion of this feature:

#99633 (comment)

Linked PRs

• gh-124872: Move PyThreadState to first argument for consistency #124774
• gh-124872: Refine contextvars documentation #124773
• gh-124872: Back up exception before calling PyContext_WatchCallback #124741
• gh-124872: Change PyContext_WatchCallback to take PyObject #124737
• gh-124872: Replace enter/exit events with "switched" #124776
• [3.13] gh-124872: Refine contextvars documentation (GH-124773) #125233
• gh-124872: Rename blurb file to reference the correct issue #125285
• gh-124872: Replace enter/exit events with "switched" #125532
• gh-124872: Mark the thread's default context as entered #125638

[ZeroIntensity]

Generally, it's a bad idea to create several different PRs for the same issue -- you can put all the needed changes in the same PR.

[rhansen]

Apologies—I was under the impression that this project squashes pull requests when merging, which I don't want (the commits are intended to be separate). I can combine them into a single PR if that would be preferred.

[ZeroIntensity]

We do squash, but why do you want the commits to be separate? It's harder to review and backport across multiple PRs

[rhansen]

I'm following the Python Developer's Guide:

• Do not fix more than one issue in the same commit (except, of course, if one code change fixes all of them).
• Do not do cosmetic changes to unrelated code in the same commit as some feature/bugfix.

Each of these PRs is focused on only one thing, and they are independent from each other (but all support the overall goal of this issue).

In general, it's bad practice to have large monolithic commits that do multiple separable things. Small, focused, and independent commits have several advantages:

• They are better for locating the root cause of a bug by bisecting.
• They are easier to cherry-pick and rebase (less likely to have merge conflicts).
• They are easier to revert if a bug is found.
• Most importantly, they are easier to understand both by current reviewers and by future code archaeologists trying to understand past design decisions.

I can put all of the commits in a single pull request, but if they are squashed then much will be lost.

[ZeroIntensity]

Each of these PRs is focused on only one thing, and they are independent from each other (but all support the overall goal of this issue).

In that case, it's better to make a new issue for each PR (I don't think it's worth doing so at this point, but this is just for future reference). If they really are all "independent", then having all the PRs under one issue makes this more difficult for us. If, for example, one PR gets merged while another gets shot down, what do we do about this issue? Does it stay closed or open?

• Most importantly, they are easier to understand both by current reviewers and by future code archaeologists trying to understand past design decisions.

Eh, looking through a tree of "related yet independent" PRs is arguably more difficult for people trying to understand design decisions. With one PR, all the reviews are in one place. For example, #124774 looks like a cosmetic change -- it would honestly fit better in a larger PR, where there's lots of other refactoring going on (and also easier to review without being brushed off as cosmetic).

I can put all of the commits in a single pull request, but if they are squashed then much will be lost.

GitHub saves the commits for us :)

In terms of bisecting, we can always checkout the branch that got merged and bisect to individual commits on it. But in my experience, we haven't ever needed to do that. Once we've found a bad commit, it's typically easy to figure out what went wrong without doing any extra micro-bisections.