Keep scrapyd running

Question

I have scrapy and scrapyd installed on a debian machine. I log in to this server using a ssh-tunnel. I then start scrapyd by going: scrapyd

Scrapyd starts up fine and I then open up another ssh-tunnel to the server and schedule my spider with: curl localhost:6800/schedule.json -d project=myproject -d spider=myspider

The spider runs nicely and everything is fine.

The problem is that scrapyd stops running when I quit the session where I started up scrapyd. This prevents me from using cron to schdedule spiders with scrapyd since scrapyd isn't running when the cronjob is launched.

My simple question is: How do I keep scrapyd running so that it doesn't shut down when I quit the ssh session.

Answer 1

Run it in a screen session:

$ screen
$ scrapyd

# hit ctrl-a, then d to detach from that screen

$ screen -r # to re-attach to your scrapyd process

Answer 2

You might consider launching scrapyd with supervisor.

And there is a good .conf script available as a gist here: https://github.com/JallyHe/scrapyd/blob/master/supervisord.conf

Answer 3

How about ? $ sudo service scrapyd start