Use a callback with a weak reference to WeakValueDictionary

[nbruin]

Following http://bugs.python.org/issue417795, we should probably only reference the dict weakly from the callback object to avoid unnecessary circular references.

As we found in #15367 comment:39, if an object C involved in a reference cycle becomes unreachable as a side effect of circular GC (e.g., a callback somewhere), the object C will only be collected by GC in the next round. A non-circularly referenced object C would have its reference count hit 0 and would be collected immediately. That is already a good reason to try and avoid circular references, even now that Python does clean them up eventually.

See also #13394 comment:19 (reasoning the validity of using a strongly referencing closure should be fine in theory -- this didn't take into account the practical consideration above).

CC: @simon-king-jena

Component: memleak

Author: Nils Bruin

Branch/Commit: u/SimonKing/ticket/15432 @ 13112a9

Reviewer: Simon King

Issue created by migration from https://trac.sagemath.org/ticket/15432

[nbruin]

Branch: u/nbruin/ticket/15432

[nbruin]

New commits:

[9bb9241](https://github.com/sagemath/sagetrac-mirror/commit/9bb9241)

trac #15432: use a callback class object holding a weak reference for sage.misc.weak_dict.WeakValueDictionary

[nbruin]

Commit: 9bb9241

[simon-king-jena]

comment:3

Just to make sure: Is there a dependency for this ticket? It seems that we need #13394 (which is merged). But #15367 is orthogonal, isn't it?

[nbruin]

comment:4

Replying to @simon-king-jena:

Just to make sure: Is there a dependency for this ticket? It seems that we need #13394 (which is merged). But #15367 is orthogonal, isn't it?

The branch here descends directly from master, which recently had 5.13beta3 merged. Therefore, as posted, this ticket has no dependencies. Since it only touches one file, that isn't even present in the branch for #15367 (its branch split off before 5.13beta3 merger), I expect no merge problems with that ticket.

I actually have a bit of trouble coming up with an example that shows a difference in behaviour for WeakValueDictionary (before this ticket it's not weakly refererrable, making examples particularly tricky). With the following code:

%cpaste
import gc
from sage.misc.weak_dict import WeakValueDictionary
from sage.structure.coerce_dict import MonoDict
def Nobj(t):
    return len([1 for o in gc.get_objects() if isinstance(o,t)])
def TestTcollect(T,args=()):
    N=Nobj(T)
    print "Number of T-objects before we start:",N
    L=[T(*args) for i in range(100)]
    M=MonoDict(11)
    for i in range(len(L)-1):
        M[L[i]]=L[i+1]
    Nnew=Nobj(T)
    print "Number of T-objects after list construction:",Nnew
    print "Throwing away references:"
    del L
    Nnew=Nobj(T)
    if Nnew == N:
        print "no GC required to get rid of objects";
    j=0
    while Nnew>N:
        j+=1
        _=gc.collect()
        Ntemp=Nobj(T)
        if Ntemp >= Nnew:
            raise RuntimeError("we're not collecting at all!")
        Nnew=Ntemp
    print "number of collects required:",j
class dd(dict):
    "a weakly reffable and easily recognized dict class"
    pass
class selfref(object):
    def __init__(self):
        self.D=self
--

I'm getting with a self-referencing class:

sage: TestTcollect(selfref)
Number of T-objects before we start: 0
Number of T-objects after list construction: 100
Throwing away references:
number of collects required: 100

The test doesn't work for dict itself (they have their own freelist etc. so
the object count you get back can't be trusted), but we can do it on a subclass:

sage: TestTcollect(dd)
Number of T-objects before we start: 0
Number of T-objects after list construction: 100
Throwing away references:
no GC required to get rid of objects
number of collects required: 0

For this patched WeakValueDictionary we're still getting problematic behaviour:

sage: TestTcollect(sage.misc.weak_dict.WeakValueDictionary)
Number of T-objects before we start: 29
Number of T-objects after list construction: 129
Throwing away references:
number of collects required: 100

Whereas python's own is clean:

sage: TestTcollect(weakref.WeakValueDictionary)
Number of T-objects before we start: 3
Number of T-objects after list construction: 103
Throwing away references:
no GC required to get rid of objects
number of collects required: 0

As is (at least our old) TripleDict:

sage: TestTcollect(sage.structure.coerce_dict.TripleDict,(11,))
Number of T-objects before we start: 180
Number of T-objects after list construction: 280
Throwing away references:
no GC required to get rid of objects
number of collects required: 0

So my guess is that we've overlooked another self reference in
WeakValueDictionary.

Note that our previous sage.misc.weak_dict.WeakValueDictionary wasn't weakly
referrable so this test doesn't apply in that case.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

[8b70d1d](https://github.com/sagemath/sagetrac-mirror/commit/8b70d1d)

trac #15432: also equip iteration context with a weak reference

[sagetrac-git]

Changed commit from 9bb9241 to 8b70d1d

[nbruin]

comment:6

And of course there is another circular reference from the _IterationContext. Making that weak as well makes the thing collectable, so with the new branch we get that TestTcollect works as desired.

Note that if you have objects chained by keys in a WeakValueDictionary, immediate collectibility of the chain could depend on the WeakValueDictionary not keeping circular references to itself.

So, I think this patch could have real effect of memory footprint in sage code.

[sagetrac-git]

Changed commit from 8b70d1d to 4ac6866

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

[4ac6866](https://github.com/sagemath/sagetrac-mirror/commit/4ac6866)

trac #15432: Code context manager in iterator via try/finally. This saves an entire class and some reference management.Z