Deploying Scrapy Spiders using a Twisted Server

Question

I have +20 scrapy crawlers that I want to deploy manually from a browser webpage. In order to achieve this, I have created a simple twisted server that executes in a shell process the following commands:

scrapyd-deploy default -p $project
curl http://127.0.0.1:6800/schedule.json -d project=$project -d spider=$spider

These commands are executed in twisted using utils.getProcessOutput(scriptname). The two previous commands are inside the script given as a parameter.

When trying to execute the twisted server with twistd -y <server.py>, it gives the following error: [Failure instance: Traceback (failure with no frames): : got stderr: 'Packing version 1399464111\n' ].

Here is the twisted server's code:

#/usr/bin/python
from twisted.internet import utils, reactor
from twisted.web import server, resource
from twisted.application import internet, service

class CrawlerResource(resource.Resource):
    isLeaf = True
    script = "./script2.sh"

    def render_GET(self, request):
        request.write("<pre>\n")
        deferred = utils.getProcessOutput(self.script)
        deferred.addCallback(lambda s: (request.write(s+"success!\n"), request.finish()))
        deferred.addErrback(lambda s: (request.write(str(s)), request.finish()))

        return server.NOT_DONE_YET



# factory: create a protocol for each connection
resource = CrawlerResource()
factory = server.Site(resource)

# application & service: run server in the background as a service
application = service.Application("Crawlers deployer")
service = internet.TCPServer(8000, factory)
service.setServiceParent(application)

What's causing this error (it isnt very verbose)?

Answer 1

Bytes on the child process's standard error stream are causing this. getProcessOutput treats any standard error output as an error and fails the Deferred.

You might fix this by passing errortoo=True to getProcessOutput. This makes it mix stdout and stderr together in the result instead of treating data on stderr as an error.

You can read about this behavior in the getProcessOutput API documentation.