Twisted reactor starting multiple times in a single program?

Question

Is it possible to start the reactor more than once in the same program? Suppose you wanted to encapsulate twisted functionality inside a method, for API purposes.

For example, mymodule.py looks like this:

  1 from twisted.web.client import getPage
  2 from twisted.internet import reactor
  3 
  4 def _result(r):
  5     print r
  6     reactor.stop()
  7 
  8 def _error(e):
  9     print e
 10     reactor.stop()
 11 
 12 def getGoogle():
 13     d = getPage('http://www.google.com')
 14     d.addCallbacks(_result, _error)
 15     reactor.run()
 16 
 17 def getYahoo():
 18     d = getPage('http://www.yahoo.com')
 19     d.addCallbacks(_result, _error)
 20     reactor.run()
 21 

main.py looks like this:

  1 import mymodule
  2 
  3 getGoogle()
  4 getYahoo()

Answer 1

Here's another way to organize your code, exploiting the single-threaded nature of Twisted: queue up all the urls you want to process, kick off the reactor, and decrement a counter when each request completes. When the counter reaches zero, stop the reactor which will return the results:

from twisted.web.client import getPage
from twisted.internet import reactor

class Getter(object):

    def __init__(self):
        self._sequence = 0
        self._results = []
        self._errors = []

    def add(self, url):
        d = getPage(url)
        d.addCallbacks(self._on_success, self._on_error)
        d.addCallback(self._on_finish)
        self._sequence += 1

    def _on_finish(self, *narg):
        self._sequence -= 1
        if not self._sequence:
            reactor.stop()

    _on_success = lambda self, *res: self._results.append(res)
    _on_error = lambda self, *err: self._errors.append(err)

    def run(self):
        reactor.run()
        return self._results, self._errors

g = Getter()
for url in ('http://www.google.com', 'http://www.yahoo.com', 'idontexist'):
    g.add(url)
results, errors = g.run()
print results
print errors

Answer 2

A more straightforward solution, which doesn't require you to manage a counter:

from twisted.internet import reactor, defer
from twisted.web.client import getPage

def printPage(page):
    print page

def printError(err):
    print err

urls = ['http://www.google.com',
        'http://www.example.com']

jobs = []
for url in urls:
    jobs.append(getPage(url).addCallbacks(printPage,
                                          printError))

def done(ignored):
    reactor.stop()
defer.gatherResults(jobs).addCallback(done)

reactor.run()

You should take a look here at what's provided by the deferred API because it will save you a lot of time and make your code easier to debug.