-
-
Notifications
You must be signed in to change notification settings - Fork 32k
bpo-42972: Fully implement GC protocol for functools keywrapper and partial types #26363
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
ce657de
to
f6e6676
Compare
f6e6676
to
6d7cdaf
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your update :)
Thank you so much for reviewing and pointing out the weakness in the LRU cache :) |
Modules/_functoolsmodule.c
Outdated
Py_VISIT(pto->args); | ||
Py_VISIT(pto->kw); | ||
Py_VISIT(pto->dict); | ||
Py_VISIT(Py_TYPE(pto)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would prefer to always start by visiting the type in all traverse functions. So same remark for other traverse functions.
I like to look into partialobject structure and visit them by their definition order. Technically, ob_type is the first one :-D
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, I'll apply to all remaining PR's :)
Modules/_functoolsmodule.c
Outdated
{Py_tp_methods, partial_methods}, | ||
{Py_tp_members, partial_memberlist}, | ||
{Py_tp_getset, partial_getsetlist}, | ||
{Py_tp_new, partial_new}, | ||
{Py_tp_free, PyObject_GC_Del}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be possible to keep it and maybe only change that in 3.11 (in a separated PR)? Type inheritance is more complex than what it looks. See https://bugs.python.org/issue43770 for examples of bad surprised that I got when trying to use the default implementation of tp_getattro and tp_setattro.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep!
Modules/_functoolsmodule.c
Outdated
PyObject_Free(ko); | ||
PyObject_GC_UnTrack(ko); | ||
(void)keyobject_clear(ko); | ||
PyObject_GC_Del(ko); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO here it's ok to hardcode PyObject_GC_Del(), but in general I would suggest to call tp->tp_free(ko);
in case tp_alloc/tp_free is overriden in a subclass. To make the code more consistent, I would suggest to always call tp_free in dealloc functions. What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree :) Consistency FTW
This reverts commit 6d7cdaf.
@pablogsal, @shihai1991, @methane: I've reverted the LRU cache optimisations as per @vstinner's request. |
Modules/_functoolsmodule.c
Outdated
Py_VISIT(link->result); | ||
Py_VISIT(Py_TYPE(link)); | ||
return 0; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A lru_list_elem_clear() function would also be needed to implement the GC protocol, no?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A lru_list_elem_clear() function would also be needed to implement the GC protocol, no?
I was under the impression traverse was enough. I may be wrong.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Without clear, the ref count cannot each zero, and the GC cannot break cycles: https://devguide.python.org/garbage_collector/#destroying-unreachable-objects
cc @pablogsal
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
_lru_list_elem
is not a standalone type. It is an internal of _lru_cache_wrapper
.
lru_cache_tp_traverse()
visitslru_list_elem
.lru_cache_tp_clear()
clearslru_list_elem
.
That's why we can keep _lru_list_elem
non-GC type.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO, Inada's #26416 is acceptable.
If you want to hide the Right now, the
And the current code for it in
But you still need to add Since the lru_cache is way more complicated, I suggest to restrict this PR to functools.KeyWrapper and partial type, and write a second PR just for the lru_cache type. |
This reverts commit fc39917.
Reverting fc39917 should be sufficient; no need to open a new PR. UPDATE: The PR now adds GC support to the KeyWrapper and partial types, and modifies |
Since _lru_list_elem is not GC-tracked type,
Current PR breaks circular reference correctly. See #26363 (comment) Since _lru_list_elem is hidden and immutable now, it is OK to remove |
It's not immutable; only static types and heap types with the |
@methane: "Since _lru_list_elem is not GC-tracked type, gc.get_objects() don't leak _lru_list_elem regardless lru_cache_tp_traverse() visit _lru_list_elem or not." It doesn't matter if the type implements the protocol or not. As soon as there is a Py_VISIT() call on it, it is exposed in gc.get_objects(). I tested the current PR (commit 6313247, Revert "Revert LRU cache element optimisation"):
Example:
Here is the hidden functools._lru_list_elem type. |
@erlend-aasland: Can you please leave LRU unchanged in this PR and create a new PR just for LRU? The LRU problem sounds complicated and IMO it deserves its own change. |
Of course. Give me 2 minutes. UPDATE: Done. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks.
Thanks for your insights and reviews, everyone! |
Thanks @erlend-aasland for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10. |
GH-26424 is a backport of this pull request to the 3.10 branch. |
…artial types (pythonGH-26363) (cherry picked from commit 8994e9c) Co-authored-by: Erlend Egeberg Aasland <[email protected]>
…artial types (GH-26363) (GH-26424) (cherry picked from commit 8994e9c) Co-authored-by: Erlend Egeberg Aasland <[email protected]>
https://bugs.python.org/issue42972