Deallocation of active generators causes error messages in Stackless 3.x

[akruis]

The following small test program emits unexpected error messages:

#
# -*- coding: utf-8 -*-
#
from __future__ import print_function, absolute_import, division

import stackless
import sys
import gc

if "hard" in sys.argv:
    stackless.enable_softswitch(False)

def gen():
    try:
        stackless.schedule_remove()
        yield 1
    except:
        print("Exception in gen:", sys.exc_info(), file=sys.stderr)
        raise

def task(generator):
    for i in generator:
        pass

t = stackless.tasklet(task)(gen())
stackless.run()
assert t.paused
print("Tasklet t is paused, going to delete t", file=sys.stderr)
# must not throw or output any error messages
t = None
gc.collect()
print("gc.collect() done", file=sys.stderr)

If you call it with Stackless 3.6 and soft-switching enabled you get:

$ ./python -u Stackless/test/test_gen_gc.py 
Tasklet t is paused, going to delete t
Exception ignored in: <generator object gen at 0x7fb752002960>
ValueError: generator already executing
gc.collect() done

And without soft-switching:

$ ./python -u Stackless/test/test_gen_gc.py hard
Tasklet t is paused, going to delete t
gc.collect() done
Exception in gen: (<class 'TaskletExit'>, TaskletExit(), <traceback object at 0x7fe135189288>)
gc:0: ResourceWarning: gc: 1 uncollectable objects at shutdown; use gc.set_debug(gc.DEBUG_UNCOLLECTABLE) to list them

With Stackless 2.7 the program does not output anything unexpected.

The observed behaviour present in Stackless 3.4 and up. Stackless 3.3 shows the hard-switching behaviour for soft switching too. This change in behaviour is obviously related to PEP 442.

Expected behaviour

If a generator is executing but it's tasklet is paused, then it is not ok to close the generator (or send anything into it) without activating its tasklet. Therefore the current behaviour is not OK. This should be fixed.

[kristjanvalur]

Interesting. So, there is an attempt made to send something into a generator on a paused tasklet (the generator is in a 'running' state, hasn't even yielded).
Exactly what do we expect to happen? A TaskletExit exception to be raised, or just everything to be collected without any finalization? (I'm getting a bit rusty in this as time goes by :)

[akruis]

Exactly what do we expect to happen?

That's a really good question. I'll have to think about it.

[akruis]

The root cause of the problem are different life cycles of tasklets and generators. A soft switched paused and restorable tasklet can be collected without any finalisation. Stackless considers such a tasklet as "data".

A not-running generator (= a generator, that has yielded) is similar to a paused and restorable tasklet. But starting with Python 3, C-Python always finalises a generator. The finaliser closes the generator (= sends a GeneratorExit exception into the generator). That's remarkably similar to the tasklet finaliser, just the exception is different.

We could modify the tasklet finaliser to always kill a living tasklets. I just mad an experimental version on top of pull request #194 and it works just fine. The generator receives a TaskletExit and errors out. (I like this modification, it makes the behaviour much simpler to understand and similar to the behaviour of generators.) The downside is a change in the behaviour of Stackless. And it does not solve the root cause: if I unbind the tasklet before releasing it (insert a t.bind(None) just before t = None) the message "ValueError: generator already executing" is back.

Because it is a normal pattern to unbind a tasklet after pickling the tasklet, a fix should work for this situation too. This requires a patch of the generator finaliser function _PyGen_Finalize.

[akruis]

Fix is in pull request #195

[akruis]

After studying the finalizers of async generators I decided to modify the behaviour of gen_close(). Now it silently skips sending GeneratorExit to a running generator. This is the documented behaviour of the method close() of these objects.

I merged pull request #195.

[kristjanvalur]

async generators, are they different from regular generators? I must confess that I have become a bit lost in what Python has been doing with async and all that....
Anyway, pickling and throwing away tasklets seems to be a very valid use case and should not raise an exception. So all of this looks good to me.