Rails: Can I run backgrounds jobs in a different server?

Question

Is it possible to host the application in one server and queue jobs in another server?

Possible examples:

1. Two different EC2 instances, one with the main server and the second with the queueing service.

2. Host the app in Heroku and use an EC2 instance with the queueing service

Is that possible?

Thanks

Answer 1

Yes, definitely. We have delayed_job set up that way where I work.

There are a couple of requirements for it to work:

1. The servers have to have synced clocks. This is usually not a problem as long as the server timezones are all set to the same.
2. The servers all have to access the same database.

To do it, you simply have the same application on both (or all, if more than two) servers, and start workers on whichever server you want to process jobs. Either server can still queue jobs, but only the one(s) with workers running will actually process them.

For example, we have one interface server, a db server and several worker servers. The interface server serves the application via Apache/Passenger, connecting the Rails application to the db server. The workers have the same application, though Apache isn't running and you can't access the application through http. They do, on the other hand, have delayed_jobs workers running. In a common scenario, the interface server queues up jobs in the db, and the worker servers process them.

One word of caution: If you're relying on physical files in your application (attachments, log files, downloaded XML or anything else), you'll most likely need a solution like S3 to keep those files. The reason for this is that the individual servers might not have the actual files. An example of this is if your user were to upload their profile picture on your web-facing server, the files would likely be stored on that server. If you then have another server to resize the profile pictures, the image wouldn't exist on the worker server.

Answer 2

Just to offer another viable option: you can use a new worker service like IronWorker that relies completely on an elastic farm of cloud servers inside EC2.

This way you can queue/schedule jobs to run and they will parallelize across tons of threads spanning multiple servers - all without worrying about the infrastructure.

Same deal with the database though - it needs to be accessible from the outside.

Full disclosure: I helped build IW.