bpo-42972: Fully implement GC protocol for functools keywrapper and partial types #26363

erlend-aasland · 2021-05-25T20:14:03Z

https://bugs.python.org/issue42972

Modules/_functoolsmodule.c

shihai1991

Thanks for your update :)

erlend-aasland · 2021-05-27T05:44:16Z

Thanks for your update :)

Thank you so much for reviewing and pointing out the weakness in the LRU cache :)

vstinner · 2021-05-27T13:51:51Z

Modules/_functoolsmodule.c

+    Py_VISIT(pto->args);
+    Py_VISIT(pto->kw);
+    Py_VISIT(pto->dict);
+    Py_VISIT(Py_TYPE(pto));


I would prefer to always start by visiting the type in all traverse functions. So same remark for other traverse functions.

I like to look into partialobject structure and visit them by their definition order. Technically, ob_type is the first one :-D

Sure, I'll apply to all remaining PR's :)

vstinner · 2021-05-27T14:04:30Z

Modules/_functoolsmodule.c

    {Py_tp_methods, partial_methods},
    {Py_tp_members, partial_memberlist},
    {Py_tp_getset, partial_getsetlist},
    {Py_tp_new, partial_new},
-    {Py_tp_free, PyObject_GC_Del},


Would it be possible to keep it and maybe only change that in 3.11 (in a separated PR)? Type inheritance is more complex than what it looks. See https://bugs.python.org/issue43770 for examples of bad surprised that I got when trying to use the default implementation of tp_getattro and tp_setattro.

vstinner · 2021-05-27T14:07:24Z

Modules/_functoolsmodule.c

-    PyObject_Free(ko);
+    PyObject_GC_UnTrack(ko);
+    (void)keyobject_clear(ko);
+    PyObject_GC_Del(ko);


IMO here it's ok to hardcode PyObject_GC_Del(), but in general I would suggest to call tp->tp_free(ko); in case tp_alloc/tp_free is overriden in a subclass. To make the code more consistent, I would suggest to always call tp_free in dealloc functions. What do you think?

I agree :) Consistency FTW

Modules/_functoolsmodule.c

This reverts commit 6d7cdaf.

erlend-aasland · 2021-05-27T18:31:56Z

@pablogsal, @shihai1991, @methane: I've reverted the LRU cache optimisations as per @vstinner's request.

vstinner · 2021-05-27T21:04:17Z

Modules/_functoolsmodule.c

+    Py_VISIT(link->result);
+    Py_VISIT(Py_TYPE(link));
+    return 0;
+}


A lru_list_elem_clear() function would also be needed to implement the GC protocol, no?

A lru_list_elem_clear() function would also be needed to implement the GC protocol, no?

I was under the impression traverse was enough. I may be wrong.

Without clear, the ref count cannot each zero, and the GC cannot break cycles: https://devguide.python.org/garbage_collector/#destroying-unreachable-objects

cc @pablogsal

_lru_list_elem is not a standalone type. It is an internal of _lru_cache_wrapper.

lru_cache_tp_traverse() visits lru_list_elem.

lru_cache_tp_clear() clears lru_list_elem.

That's why we can keep _lru_list_elem non-GC type.

IMO, Inada's #26416 is acceptable.

vstinner · 2021-05-28T07:21:44Z

If you want to hide the functools._lru_list_elem type, the lru_cache_tp_traverse() function must not traverse its type. Otherwise, there is a risk that the type is discovered using gc.get_objects() or a similar function.

Right now, the functools._lru_list_elem type is hidden in the main branch and it doesn't implement the GC protocol. In this case, the current code for it in lru_cache_tp_traverse() in ok:

    lru_list_elem *link = self->root.next;
    while (link != &self->root) {
        lru_list_elem *next = link->next;
        Py_VISIT(link->key);
        Py_VISIT(link->result);
        link = next;
    }

And the current code for it in lru_cache_cache_clear() is also ok:

    lru_cache_clear_list(list);

But you still need to add Py_VISIT(Py_TYPE(self)); in lru_cache_tp_traverse().

Since the lru_cache is way more complicated, I suggest to restrict this PR to functools.KeyWrapper and partial type, and write a second PR just for the lru_cache type.

This reverts commit fc39917.

erlend-aasland · 2021-05-28T07:48:06Z

Since the lru_cache is way more complicated, I suggest to restrict this PR to functools.KeyWrapper and partial type, and write a second PR just for the lru_cache type.

Reverting fc39917 should be sufficient; no need to open a new PR.

UPDATE: The PR now adds GC support to the KeyWrapper and partial types, and modifies lru_cache_tp_traverse to visit the LRU cache type and the LRU cache element type.

methane · 2021-05-28T07:52:36Z

If you want to hide the functools._lru_list_elem type, the lru_cache_tp_traverse() function must not traverse its type. Otherwise, there is a risk that the type is discovered using gc.get_objects() or a similar function.

Since _lru_list_elem is not GC-tracked type, gc.get_objects() don't leak _lru_list_elem regardless lru_cache_tp_traverse() visit _lru_list_elem or not.

Right now, the functools._lru_list_elem type is hidden in the main branch and it doesn't implement the GC protocol. In this case, the current code for it in lru_cache_tp_traverse() in ok:

Current PR breaks circular reference correctly. See #26363 (comment)

Since _lru_list_elem is hidden and immutable now, it is OK to remove Py_VISIT(Py_TYPE(link)); if we are sure about _lru_list_elem_type don't create circular reference.
But I am not sure about it. So I feel current PR is the best. It breaks circular reference correctly anyway.

erlend-aasland · 2021-05-28T07:55:05Z

Since _lru_list_elem is hidden and immutable now [...]

It's not immutable; only static types and heap types with the Py_TPFLAGS_IMMUTABLETYPE flag are immutable.

vstinner · 2021-05-28T08:13:09Z

@methane: "Since _lru_list_elem is not GC-tracked type, gc.get_objects() don't leak _lru_list_elem regardless lru_cache_tp_traverse() visit _lru_list_elem or not."

It doesn't matter if the type implements the protocol or not. As soon as there is a Py_VISIT() call on it, it is exposed in gc.get_objects().

I tested the current PR (commit 6313247, Revert "Revert LRU cache element optimisation"):

static int
lru_cache_tp_traverse(lru_cache_object *self, visitproc visit, void *arg)
{
    ...
    lru_list_elem *link = self->root.next;
    while (link != &self->root) {
        lru_list_elem *next = link->next;
        Py_VISIT(link->key);
        Py_VISIT(link->result);
        Py_VISIT(Py_TYPE(link));  // <==== HERE
        link = next;
    }
    ...
}

Example:

$ cat x.py 
import functools

class Value:
    pass
value = Value()

@functools.lru_cache
def func():
    return value

func()

$ ./python -i x.py 
>>> import gc
>>> isinstance(int, type)
True
>>> types=[obj for obj in gc.get_objects() if isinstance(obj, type)]
>>> lru=[type for type in types if 'lru' in type.__qualname__]
>>> lru
[<class 'functools._lru_cache_wrapper'>, <class 'functools._lru_list_elem'>]

Here is the hidden functools._lru_list_elem type.

vstinner · 2021-05-28T08:13:45Z

@erlend-aasland: Can you please leave LRU unchanged in this PR and create a new PR just for LRU? The LRU problem sounds complicated and IMO it deserves its own change.

erlend-aasland · 2021-05-28T08:15:22Z

@erlend-aasland: Can you please leave LRU unchanged in this PR and create a new PR just for LRU? The LRU problem sounds complicated and IMO it deserves its own change.

Of course. Give me 2 minutes. UPDATE: Done.

vstinner

LGTM, thanks.

erlend-aasland · 2021-05-28T08:27:57Z

@vstinner @methane, see GH-26423

erlend-aasland · 2021-05-28T08:29:21Z

Thanks for your insights and reviews, everyone!

miss-islington · 2021-05-28T08:41:18Z

Thanks @erlend-aasland for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

bedevere-bot · 2021-05-28T08:41:29Z

GH-26424 is a backport of this pull request to the 3.10 branch.

…artial types (pythonGH-26363) (cherry picked from commit 8994e9c) Co-authored-by: Erlend Egeberg Aasland <[email protected]>

…artial types (GH-26363) (GH-26424) (cherry picked from commit 8994e9c) Co-authored-by: Erlend Egeberg Aasland <[email protected]>

erlend-aasland requested a review from rhettinger as a code owner May 25, 2021 20:14

the-knights-who-say-ni added the CLA signed label May 25, 2021

bedevere-bot added the awaiting review label May 25, 2021

erlend-aasland requested a review from pablogsal May 25, 2021 20:14

erlend-aasland added skip news needs backport to 3.10 only security fixes labels May 25, 2021

shihai1991 reviewed May 26, 2021

View reviewed changes

Modules/_functoolsmodule.c Outdated Show resolved Hide resolved

shihai1991 requested a review from methane May 26, 2021 10:19

methane reviewed May 26, 2021

View reviewed changes

Modules/_functoolsmodule.c Outdated Show resolved Hide resolved

erlend-aasland force-pushed the bpo-42972/functools branch 2 times, most recently from ce657de to f6e6676 Compare May 26, 2021 20:48

erlend-aasland marked this pull request as draft May 26, 2021 20:51

Erlend E. Aasland added 2 commits May 27, 2021 01:58

bpo-42972: Fully implement GC protocol for functools types

8ef6e98

Address review

6d7cdaf

erlend-aasland force-pushed the bpo-42972/functools branch from f6e6676 to 6d7cdaf Compare May 26, 2021 23:58

erlend-aasland marked this pull request as ready for review May 27, 2021 00:01

shihai1991 approved these changes May 27, 2021

View reviewed changes

bedevere-bot added awaiting core review and removed awaiting review labels May 27, 2021

Address review: visit node type for each strong ref

a15b49f

vstinner reviewed May 27, 2021

View reviewed changes

Erlend E. Aasland added 6 commits May 27, 2021 19:37

Address review: visit members in correct order

0c8df1f

Address review: revert Py_CLEAR change in lru_list_elem_dealloc

f9ac4c5

Revert LRU cache element optimisation

fc39917

This reverts commit 6d7cdaf.

Address review: revert redundant slot removal

07f2182

Address review: use tp_free

0861788

Revert lru list elem clear fn

7c2f750

vstinner reviewed May 27, 2021

View reviewed changes

Revert "Revert LRU cache element optimisation"

6313247

This reverts commit fc39917.

Restore lru_cache_tp_traverse

1539195

erlend-aasland requested a review from vstinner May 28, 2021 08:20

erlend-aasland changed the title ~~bpo-42972: Fully implement GC protocol for functools types~~ bpo-42972: Fully implement GC protocol for functools keywrapper and partial types May 28, 2021

vstinner approved these changes May 28, 2021

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels May 28, 2021

vstinner merged commit 8994e9c into python:main May 28, 2021

bedevere-bot removed the awaiting merge label May 28, 2021

bedevere-bot removed the needs backport to 3.10 only security fixes label May 28, 2021

vstinner pushed a commit that referenced this pull request May 28, 2021

bpo-42972: Fully implement GC protocol for functools keywrapper and p…

eb8ab04

…artial types (GH-26363) (GH-26424) (cherry picked from commit 8994e9c) Co-authored-by: Erlend Egeberg Aasland <[email protected]>

erlend-aasland deleted the bpo-42972/functools branch May 28, 2021 09:59

pablogsal mentioned this pull request Jul 18, 2021

[3.10] Correct the order of regen-abidump #27228

Closed

vstinner mentioned this pull request Jun 9, 2022

[C API] Heap types (PyType_FromSpec) must fully implement the GC protocol #87138

Closed

Uh oh!

bpo-42972: Fully implement GC protocol for functools keywrapper and partial types #26363

bpo-42972: Fully implement GC protocol for functools keywrapper and partial types #26363

Uh oh!

Conversation

erlend-aasland commented May 25, 2021 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shihai1991 left a comment

Choose a reason for hiding this comment

Uh oh!

erlend-aasland commented May 27, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

erlend-aasland commented May 27, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vstinner commented May 28, 2021

Uh oh!

erlend-aasland commented May 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

methane commented May 28, 2021

Uh oh!

erlend-aasland commented May 28, 2021

Uh oh!

vstinner commented May 28, 2021

Uh oh!

vstinner commented May 28, 2021

Uh oh!

erlend-aasland commented May 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

erlend-aasland commented May 28, 2021

Uh oh!

erlend-aasland commented May 28, 2021

Uh oh!

miss-islington commented May 28, 2021

Uh oh!

bedevere-bot commented May 28, 2021

Uh oh!

Uh oh!

erlend-aasland commented May 25, 2021 •

edited by bedevere-bot

Loading

erlend-aasland commented May 28, 2021 •

edited

Loading

erlend-aasland commented May 28, 2021 •

edited

Loading