Python DeferredList callback reporting success when deferreds raise error

Question

I have the following simple script:

#!/usr/bin/env python
from twisted.internet import defer             
from twisted.web.client import getPage, reactor

def success(results):
  print 'success'              

def error(results):
  print 'error'         

def finished(results):
  print 'finished', results

tasks = []

d = getPage('thiswontwork').addCallback(success).addErrback(error)
tasks.append(d)

dl = defer.DeferredList(tasks)
dl.addCallback(finished)

reactor.run()

That produces the following output:

error
finished [(True, None)]

I would expect this error task to return a false since the getPages task fails and calls it's error callback. Can anybody explain this behaviour?

Answer 1

Let me add a thought beyond what you selected as the answer:

In your question you print the parameter that your deferred(list) callback(finished) is called with, and you're noting that you would expect it to be false (because of the error happening prior)

That expectation suggests your thinking about deferreds in a way thats going to confuse you later.

Let me see if I can break this apart...

(There are a lot of details and subtly in this stuff, I'm going to try my best)

A deferred is:

1. a queue of function callbacks
2. Which, for each 'step' in the queue there is both a callback and and errback function callback stored
   • addCallbacks (note its plural) lets you add both the callback and the errback at once
   • addCallback adds a callback (and sets the errback to skip to the next entry in the queue)
   • addErrback adds a errback (and sets the callback to skip to the next entry in the queue)
   • addBoth add a function as both the errback and the callback (... which means it needs to figure out why its being called via the arg its called with

Drawing from krondo's part 7: (...Twisted docs also have a good drawing)

If a function is added to a deferred as a callback and the deferred has reached the point in its 'firing' that the step before this one return successfully, the deferred with pass the return value of the previous successfully function to the function defined in the callback.

If a function is added to a deferred as a errback and the deferred has reached the point in its 'firing' where the previous step returned a Failure object (or raised an exception, which twisted transforms into a Failure object), then the errback will be called with that Failure object. Note! if the errback doesn't return a Failure, the deferred will flip back to calling the callback chain, not the errback!

Drawing from Krondo's part 9:

While this all may seem like a bit of a brain-twister, it lets you implement things like error-recovery mid-deferred, which can be really helpful (and not all that uncommon in protocol design)

I.E: (Drawing from Krondo's part 9):

To put this all together, the error in your thought of "I would expect this error task to return a false" is that finished isn't called by any sort of error task, its called by the deferred, and on top of that It's only called on success (because its only loaded into the deferred as a callback not as an errback)

If you had loaded the finished function as both an errback and a callback (perhaps via addBoth), and you had followed the answers advice to forward the return-state of the errback via returning the Failure object, your finished function still technically wouldn't be passed False! it would receive a Failure object.

... Like I said, lots of subtly in this stuff ...

If you find any of this helpful (... or even if you don't - I'm not as good of a writer), you really should dig through Krondo's twisted introduction. I think you'll find a lot of this snaps into focus after going through that guide.

Answer 2

If you don't return the error/exception from your errback, then that error is considered to be handled. Try changing your error() function:

def error(results):
    print 'error'
    return results

This way the error (results) is returned, and then the DeferredList will see the error and call its errback as well.