-
Notifications
You must be signed in to change notification settings - Fork 245
Can't access full stack of suspended greenlet on 3.12 #388
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
OK, I dug into this a bit more. On 3.12, the linked list of It's worth noting that the current approach is not sufficient to prevent crashes, because it's still possible to get a reference to a frame that is not the innermost at the time of its greenlet's suspension:
I've developed a patch that provides a If we want to totally prevent this crash, I think we would need to walk the frame list on every greenlet suspension. There are three options I see for doing that:
Obviously none of these is particularly appealing from a performance perspective. To mitigate that impact, any of them could also be done from the Thoughts? |
Thanks for your report and detailed analysis! It's much appreciated. What follows are some uncoordinated, half-baked thoughts I'm tossing out in between meetings: I suspect at least some of the problem predates 3.12; 3.11 is when CPython started storing frames on the C stack (IIRC; it could have been 3.10). I'm looking to reduce switching overhead, not increase it. We've fallen behind in that area in recent versions of CPython. If the potential for crashing predates 3.12 (I'm not totally sure there off the top of my head), I'm not recalling any crash reports related to accessing previously captured frames from switched out greenlets, so that doesn't seem to be a common pattern. Unfortunately for us, the direction CPython development is going looks to be imposing more and more limits on what greenlet can do in the name of interpreter performance (which is a perfectly reasonable tradeoff!); for example, tracing is of limited utility under 3.12 because the control information for that too has moved to inaccessible places. So I'm afraid we're going to have to get used to compromises. Or just wait for the no-GIL version of CPython 3.13 and switch to threads 😄 That all in mind, I think I'd be more inclined to do something like make |
No, this is new in 3.12. The PyInterpreterFrame structure existed in 3.11 but none of them were on the C stack; in 3.11 all interpreter frames were contained either in a generator object, frame object, or on the per-thread frame stack. The change that created problems for us is python/cpython#96319 and its rationale is described (lightly) in python/cpython#96421 .
Sure, that makes sense to me.
I think that not being able to introspect a suspended greenlet's stack is a pretty major usability problem for debugging tools. Would you be willing to consider a solution where the ugly stuff is in the
I think that ship has sailed, honestly -- the greenlet package is tied to CPython implementation details for better or for worse. It's reasonable to decide that some feature is no longer supportable, at which point I agree that it's better to fail cleanly than to produce an incorrect result; but I don't think we're at that point yet wrt frame extraction. There might also be a palatable upstream change that would restore greenlet's ability to extract a suspended greenlet's stack. I'll file an issue about that and see what the CPython maintainers say. |
No response from CPython, but I think we can solve this satisfactorily on our end - see attached PR. It's actually slightly faster than the 3.12 status quo if you don't access gr_frame. |
Bumps [greenlet](https://github.com/python-greenlet/greenlet) from 3.0.0 to 3.0.3. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/python-greenlet/greenlet/blob/master/CHANGES.rst">greenlet's changelog</a>.</em></p> <blockquote> <h1>3.0.3 (2023-12-21)</h1> <ul> <li>Python 3.12: Restore the full ability to walk the stack of a suspended greenlet; previously only the innermost frame was exposed. See <code>issue 388 <https://github.com/python-greenlet/greenlet/issues/388></code><em>. Fix by Joshua Oreman in <code>PR 393 <https://github.com/python-greenlet/greenlet/pull/393/></code></em>.</li> </ul> <h1>3.0.2 (2023-12-08)</h1> <ul> <li>Packaging: Add a minimal <code>pyproject.toml</code> to sdists.</li> <li>Packaging: Various updates to macOS wheels.</li> <li>Fix a test case on Arm32. Note that this is not a supported platform (there is no CI for it) and support is best effort; there may be other issues lurking. See <code>issue 385 <https://github.com/python-greenlet/greenlet/issues/385></code>_</li> </ul> <h1>3.0.1 (2023-10-25)</h1> <ul> <li>Fix a potential crash on Python 3.8 at interpreter shutdown time. This was a regression from earlier 3.0.x releases. Reported by Matt Wozniski in <code>issue 376 <https://github.com/python-greenlet/greenlet/issues/376></code>_.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/python-greenlet/greenlet/commit/ea4bc2776c75429a577e539389fee40ad9e46707"><code>ea4bc27</code></a> Preparing release 3.0.3</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/7694880b128bfa54ea0ece59974695a06af09931"><code>7694880</code></a> Make doctests work on 3.7 again, which doesn't have importlib.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/073b1e1cf4ec25a8b90a534176bf79336d87689e"><code>073b1e1</code></a> Linting. Add linting to CI.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/9e73b59eadb68017c25e8fabf0b9dba2eef6f150"><code>9e73b59</code></a> Docs: Update from the old default theme to furo.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/5f4b4bbc098330b87211405ef50e276325c15a53"><code>5f4b4bb</code></a> Py3.12: Always expose greenlet frames on a switch.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/13148f9f975924da18bef14054a558160e13369c"><code>13148f9</code></a> Update comment that was still referring to a different, less-robust approach ...</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/b5f9d232c401d8fba8710778ad85b29a41ec3924"><code>b5f9d23</code></a> Restore stack introspection ability on 3.12</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/edbdda27ab1983ac157b588dd0c04816cb31b0ea"><code>edbdda2</code></a> Back to development: 3.0.3</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/719ea473c61e9934cb75ebd04a52dda32a030863"><code>719ea47</code></a> Preparing release 3.0.2</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/2c0793c1b35e785d4353d4c6ce17cde2137da1c4"><code>2c0793c</code></a> Add change note about macOS wheels.</li> <li>Additional commits viewable in <a href="https://github.com/python-greenlet/greenlet/compare/3.0.0...3.0.3">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>
Bumps [greenlet](https://github.com/python-greenlet/greenlet) from 3.0.1 to 3.0.3. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/python-greenlet/greenlet/blob/master/CHANGES.rst">greenlet's changelog</a>.</em></p> <blockquote> <h1>3.0.3 (2023-12-21)</h1> <ul> <li>Python 3.12: Restore the full ability to walk the stack of a suspended greenlet; previously only the innermost frame was exposed. See <code>issue 388 <https://github.com/python-greenlet/greenlet/issues/388></code><em>. Fix by Joshua Oreman in <code>PR 393 <https://github.com/python-greenlet/greenlet/pull/393/></code></em>.</li> </ul> <h1>3.0.2 (2023-12-08)</h1> <ul> <li>Packaging: Add a minimal <code>pyproject.toml</code> to sdists.</li> <li>Packaging: Various updates to macOS wheels.</li> <li>Fix a test case on Arm32. Note that this is not a supported platform (there is no CI for it) and support is best effort; there may be other issues lurking. See <code>issue 385 <https://github.com/python-greenlet/greenlet/issues/385></code>_</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/python-greenlet/greenlet/commit/ea4bc2776c75429a577e539389fee40ad9e46707"><code>ea4bc27</code></a> Preparing release 3.0.3</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/7694880b128bfa54ea0ece59974695a06af09931"><code>7694880</code></a> Make doctests work on 3.7 again, which doesn't have importlib.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/073b1e1cf4ec25a8b90a534176bf79336d87689e"><code>073b1e1</code></a> Linting. Add linting to CI.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/9e73b59eadb68017c25e8fabf0b9dba2eef6f150"><code>9e73b59</code></a> Docs: Update from the old default theme to furo.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/5f4b4bbc098330b87211405ef50e276325c15a53"><code>5f4b4bb</code></a> Py3.12: Always expose greenlet frames on a switch.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/13148f9f975924da18bef14054a558160e13369c"><code>13148f9</code></a> Update comment that was still referring to a different, less-robust approach ...</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/b5f9d232c401d8fba8710778ad85b29a41ec3924"><code>b5f9d23</code></a> Restore stack introspection ability on 3.12</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/edbdda27ab1983ac157b588dd0c04816cb31b0ea"><code>edbdda2</code></a> Back to development: 3.0.3</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/719ea473c61e9934cb75ebd04a52dda32a030863"><code>719ea47</code></a> Preparing release 3.0.2</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/2c0793c1b35e785d4353d4c6ce17cde2137da1c4"><code>2c0793c</code></a> Add change note about macOS wheels.</li> <li>Additional commits viewable in <a href="https://github.com/python-greenlet/greenlet/compare/3.0.1...3.0.3">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [greenlet](https://github.com/python-greenlet/greenlet) from 3.0.1 to 3.0.3. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/python-greenlet/greenlet/blob/master/CHANGES.rst">greenlet's changelog</a>.</em></p> <blockquote> <h1>3.0.3 (2023-12-21)</h1> <ul> <li>Python 3.12: Restore the full ability to walk the stack of a suspended greenlet; previously only the innermost frame was exposed. See <code>issue 388 <https://github.com/python-greenlet/greenlet/issues/388></code><em>. Fix by Joshua Oreman in <code>PR 393 <https://github.com/python-greenlet/greenlet/pull/393/></code></em>.</li> </ul> <h1>3.0.2 (2023-12-08)</h1> <ul> <li>Packaging: Add a minimal <code>pyproject.toml</code> to sdists.</li> <li>Packaging: Various updates to macOS wheels.</li> <li>Fix a test case on Arm32. Note that this is not a supported platform (there is no CI for it) and support is best effort; there may be other issues lurking. See <code>issue 385 <https://github.com/python-greenlet/greenlet/issues/385></code>_</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/python-greenlet/greenlet/commit/ea4bc2776c75429a577e539389fee40ad9e46707"><code>ea4bc27</code></a> Preparing release 3.0.3</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/7694880b128bfa54ea0ece59974695a06af09931"><code>7694880</code></a> Make doctests work on 3.7 again, which doesn't have importlib.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/073b1e1cf4ec25a8b90a534176bf79336d87689e"><code>073b1e1</code></a> Linting. Add linting to CI.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/9e73b59eadb68017c25e8fabf0b9dba2eef6f150"><code>9e73b59</code></a> Docs: Update from the old default theme to furo.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/5f4b4bbc098330b87211405ef50e276325c15a53"><code>5f4b4bb</code></a> Py3.12: Always expose greenlet frames on a switch.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/13148f9f975924da18bef14054a558160e13369c"><code>13148f9</code></a> Update comment that was still referring to a different, less-robust approach ...</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/b5f9d232c401d8fba8710778ad85b29a41ec3924"><code>b5f9d23</code></a> Restore stack introspection ability on 3.12</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/edbdda27ab1983ac157b588dd0c04816cb31b0ea"><code>edbdda2</code></a> Back to development: 3.0.3</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/719ea473c61e9934cb75ebd04a52dda32a030863"><code>719ea47</code></a> Preparing release 3.0.2</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/2c0793c1b35e785d4353d4c6ce17cde2137da1c4"><code>2c0793c</code></a> Add change note about macOS wheels.</li> <li>Additional commits viewable in <a href="https://github.com/python-greenlet/greenlet/compare/3.0.1...3.0.3">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
The
gr_frame
attribute refers to the innermost frame (the frame that calledswitch()
), but its f_back attribute points nowhere.On 3.11 I see instead:
which is what I would expect from the documentation. This looks related to some logic in
TPythonState.cpp
that moves the previous ptr into greenlet-local storage when suspending:How is greenlet introspection supposed to work on 3.12?
The text was updated successfully, but these errors were encountered: