Twisted Python pause/postpone reactor

Question

I'm pretty new to twisted, I have an HTTP client that queries a server that has rate limit, when I hit this limit the server responds with HTTP 204, so when I'm handling the response I'm doing probably something nasty, like this:

def handleResponse(r, ip):
if r.code == 204:
    print 'Got 204, sleeping'
    time.sleep(120)
    return None
else:
    jsonmap[ip] = ''
    whenFinished = twisted.internet.defer.Deferred()
    r.deliverBody(PrinterClient(whenFinished, ip))
    return whenFinished

I'm doing this because I want to pause all the tasks.

Following there are 2 behaviours that I've in my mind, either re-run the tasks that hit 204 afterwards in the same execution (don't know if it's possible) or just log the errors and re-run them afterwards in another execution of the program. Another problem that may raise is that I've set a timeout on the connection in order to cancel the deferred after a pre-defined amount of time (see the code below) if there's no response from the server

timeoutCall = reactor.callLater(60, d.cancel)
def completed(passthrough):
    if timeoutCall.active():
        timeoutCall.cancel()
    return passthrough
d.addCallback(handleResponse, ip)
d.addErrback(handleError, ip)
d.addBoth(completed)

Another problem that I may encounter is that if I'm sleeping I may hit this timeout and all my requests will be cancelled.

I hope that I've been enough precise. Thank you in advance.

Jeppo

Answer 1

Don't use time.sleep(20) in any Twisted-based code. This violates basic assumptions that any other Twisted-based code that you might be using makes.

Instead, if want to delay something by N seconds, use reactor.callLater(N, someFunction).

Once you remove the sleep calls from your program, the problem of unrelated timeouts being hit just because you've stopped the reactor from processing events will go away.

Answer 2

For anyone stumbling across this thread, it's imperative that you never call time.sleep(...); however, it is possible to create a Deferred that does nothing but sleep... which you can use to compose delays into a deferred chain:

   def make_delay_deferred(seconds, result=None):
       d = Deferred()
       reactor.callLater(seconds, d.callback, result)
       return d