Error when using twisted and greenlets

Question

I'm trying to use twisted with greenlets, so I can write synchronous looking code in twisted without using inlineCallbacks.

Here is my code:

import time, functools
from twisted.internet import reactor, threads
from twisted.internet.defer import Deferred
from functools import wraps
import greenlet

def make_async(func):
    @wraps(func)
    def wrapper(*pos, **kwds):
        d = Deferred()

        def greenlet_func():
            try:
                rc = func(*pos, **kwds)
                d.callback(rc)
            except Exception, ex:
                print ex
                d.errback(ex)

        g = greenlet.greenlet(greenlet_func)
        g.switch()

        return d
    return wrapper

def sleep(t):
    print "sleep(): greenelet:", greenlet.getcurrent()
    g = greenlet.getcurrent()
    reactor.callLater(t, g.switch)
    g.parent.switch()

def wait_one(d):
    print "wait_one(): greenelet:", greenlet.getcurrent()
    g = greenlet.getcurrent()
    active = True

    def callback(result):
        if not active:
            g.switch(result)
        else:
            reactor.callLater(0, g.switch, result)

    def errback(failure):
        if not active:
            g.throw(failure)
        else:
            reactor.callLater(0, g.throw, failure)

    d.addCallback(callback)
    d.addErrback(errback)

    active = False
    rc = g.parent.switch()
    return rc

@make_async
def inner():
    print "inner(): greenelet:", greenlet.getcurrent()

    import random, time
    interval = random.random()

    print "Sleeping for %s seconds..." % interval
    sleep(interval)
    print "done"

    return interval

@make_async
def outer():
    print "outer(): greenelet:", greenlet.getcurrent()
    print wait_one(inner())
    print "Here"

reactor.callLater(0, outer)
reactor.run()

There are 5 main parts:

• A sleep function, that starts a timer, then switches back to the parent greenlet. When the timer goes off, it switches back to the greenlet that is sleeping.
• A make_async decorator. This takes some synchronous looking code and runs it in a greenlet. IT also returns a deferred so the caller can register callbacks when the code completes.
• A wait_one function, which blocks the greenlet until the deferred being waited on resolves.
• The inner function, which (when wrapped) returns a deferred, sleeps for a random time, and then passes the time it slept for to the deferred.
• The outer function, which calls inner(), waits for it to return, then prints the return value.

When I run this code I get this output (Note the error on the last two lines):

outer(): greenelet: <greenlet.greenlet object at 0xb729cc5c>
inner(): greenelet: <greenlet.greenlet object at 0xb729ce3c>
Sleeping for 0.545666723422 seconds...
sleep(): greenelet: <greenlet.greenlet object at 0xb729ce3c>
wait_one(): greenelet: <greenlet.greenlet object at 0xb729cc5c>
done
0.545666723422
Here
Exception twisted.python.failure.Failure: <twisted.python.failure.Failure <class 'greenlet.GreenletExit'>> in <greenlet.greenlet object at 0xb729ce3c> ignored
GreenletExit did not kill <greenlet.greenlet object at 0xb729ce3c>

Doing a bit of research I've found that:

• The last line is logged by greenlet.c
• The previous line is logged by python itself, as it's ignoring an exception raised in a del method.

I'm having real trouble debugging this as I can't access the GreenletExit or twisted.python.failure.Failure exceptions to get their stack traces.

Does anyone have any ideas what I'm doing wrong, or how I get debug the exceptions that are being thrown?

One other data point: If I hack wait_one() to just return immediately (and not to register anything on the deferred it is passed), the errors go away. :-/

Answer 1

Rewrite your error callback in wait_one like this:

  def errback(failure):
    ## new code
    if g.dead:
        return
    ##
    if not active:
        g.throw(failure)
    else:
        reactor.callLater(0, g.throw, failure)

If greenlet is dead (finished running), there is no point throwing exceptions in it.

Answer 2

mguijarr's answer fixed the problem, but I wanted to write up how I got into this situation.

I have three greenlets:

• {main} that's runing the reactor.
• {outer} that's running outer().
• {inner} that's rrunning inner().

When the sleep finishes the {main} switches to {inner} which switches to {outer}. Outer then returns and raises GreenletExit in {inner}. This propogates back to twisted. It sees an exception being raised from callback(), and so invokes errback(). This tries to throw the exception into {outer} (which has already exited), and I hit the error.