Issue with downloadPage callback function

Question

I have written the following python script:

from twisted.internet import defer             
from twisted.web.client import getPage, downloadPage, reactor
import tempfile

def success(results):
  print 'success'   

def error(results):
  print 'error', results
  reactor.stop()

tmpfilename = tempfile.mkstemp()
downloadPage('http://www.google.com', tmpfilename).addCallback(success).addErrback(error)

reactor.run()

And am getting the following error:

Unhandled Error
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/twisted/python/log.py", line 88, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/usr/local/lib/python2.7/site-packages/twisted/python/log.py", line 73, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/usr/local/lib/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/selectreactor.py", line 151, in _doReadOrWrite
    why = getattr(selectable, method)()
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/tcp.py", line 215, in doRead
    return self._dataReceived(data)
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/tcp.py", line 221, in _dataReceived
    rval = self.protocol.dataReceived(data)
  File "/usr/local/lib/python2.7/site-packages/twisted/protocols/basic.py", line 578, in dataReceived
    why = self.rawDataReceived(data)
  File "/usr/local/lib/python2.7/site-packages/twisted/web/http.py", line 518, in rawDataReceived
    self.handleResponsePart(data)
  File "/usr/local/lib/python2.7/site-packages/twisted/web/client.py", line 249, in handleResponsePart
    self.factory.pagePart(data)
  File "/usr/local/lib/python2.7/site-packages/twisted/web/client.py", line 504, in pagePart
    self.file.write(data)
exceptions.AttributeError: 'tuple' object has no attribute 'write'
Unhandled Error
Traceback (most recent call last):
  File "poc.py", line 16, in <module>
    reactor.run()
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1192, in run
    self.mainLoop()
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1204, in mainLoop
    self.doIteration(t)
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/selectreactor.py", line 145, in doSelect
    _logrun(selectable, _drdw, selectable, method)
--- <exception caught here> ---
  File "/usr/local/lib/python2.7/site-packages/twisted/python/log.py", line 88, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/usr/local/lib/python2.7/site-packages/twisted/python/log.py", line 73, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/usr/local/lib/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/selectreactor.py", line 156, in _doReadOrWrite
    self._disconnectSelectable(selectable, why, method=="doRead")
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/posixbase.py", line 263, in _disconnectSelectable
    selectable.connectionLost(failure.Failure(why))
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/tcp.py", line 485, in connectionLost
    self._commonConnection.connectionLost(self, reason)
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/tcp.py", line 299, in connectionLost
    protocol.connectionLost(reason)
  File "/usr/local/lib/python2.7/site-packages/twisted/web/client.py", line 198, in connectionLost
    http.HTTPClient.connectionLost(self, reason)
  File "/usr/local/lib/python2.7/site-packages/twisted/web/http.py", line 472, in connectionLost
    self.handleResponseEnd()
  File "/usr/local/lib/python2.7/site-packages/twisted/web/client.py", line 258, in handleResponseEnd
    self.factory.pageEnd()
  File "/usr/local/lib/python2.7/site-packages/twisted/web/client.py", line 531, in pageEnd
    self.file.close()
exceptions.AttributeError: 'tuple' object has no attribute 'close'

If I change the url to something invalid it will throw the correct error callback function so it appears to be something to do with the success callback however I can't understand why.

Answer 1

After this:

tmpfilename = tempfile.mkstemp()

The value of tmpfilename is a tuple (see docs) but twisted expects a filename or file-like object.

So you could do something like:

tmpfile = tempfile.mkstemp()
tmpfilename = tmpfile[1]
downloadPage('http://www.google.com', tmpfilename).addCallback(success).addErrback(error)

which works.

But if you don't need the file to persist, I'd recommend something like:

tmpfile = tempfile.TemporaryFile()
downloadPage('http://www.google.com', tmpfile).addCallback(success).addErrback(error)

which uses the TemporaryFile() constructor giving you access to the downloaded data, but once the process has closed, the file is (for all intents and purposes) gone never to be seen again.

You could further improve this by using a context manager -- something like:

with tempfile.TemporaryFile() as tmpfile:
    downloadPage('http://www.google.com', tmpfile).addCallback(success).addErrback(error)

    # do other stuff with tmpfile

# code that no longer depends on the existence of tmpfile