When to use a message queue and when to use a cloud background-worker

Question

When would I use a message queue like ironMQ and when would i use a job processing worker like ironWorker?

I have just started researching into these two topics and I am finding it hard to distinguish between the two uses. I understand a worker is more or less a sandbox that will run a program in a different environment outside of the app server to increase the user experience. I also understand that a message queue is much like its database alternative whereby a task is added to a queue and then another server/programming listens for that task and will then process it. However, although I think I understand what they are I am having trouble distinguishing when I would use each one and why.

If I understand correctly I would use a worker for a task such as image processing. But then why can't I use a message queue for that and more importantly why not? Surely I could just have an image URL queued in ironMQ and then have another programming retrieve and process it. In my mind that seems like an extra step so I would avoid that.

A message queue seems fairly pointless to me for common tasks when a worker is available. Surely for non-intensive tasks like posting a comment I could have a worker do that?

I may have misconceived the difference between each tool and if so please set me straight. Otherwise, please help.

Answer 1

They are very closely related so I can understand the confusion. They are both queue based systems, one being a message queue, one being a task/job queue. Here's a general rule of thumb:

• You'd use IronMQ if you want to run the consumer/worker processes/servers.
• You'd use IronWorker if you want Iron.io to take care of the consumer/worker processes/servers. You also get a bunch of other features/benefits using IronWorker, not just the fact that you don't have to manage your own servers and scale them, etc.

So no, you don't need a message queue if you're using IronWorker because IronWorker is your message queue + your processing of that queue.

Not to add any confusion, but some people use IronWorker and IronMQ together too, with workers pulling messages off IronMQ. This pattern is good for very short tasks to amortize the setup/teardown of a worker (making database connections or whatever the worker has to do to setup).

Answer 2

Message queue is useful when you want to send some data to other process, part of application, or, even, different application. It works like pipe, which transfer data to other side. For example, I have 10 online shops where customers buy things. I want to process all the checkouts on one server. There are some ways to connect shops to processing center:

1. Make API on the checkouts processing center server; requires human resources to write API code, API and client on shop side must be failsafe (checkouts must be delivered to processing center).
2. Put checkouts to DB from shop server, get them on processing server; it requires access to the same DB from shops and processing center which, sometimes, is not secure enough. Note, that use of separate DB server must act as message queue. It must be scalable, so, I need to spend money for infrastructure and administrator.
3. Send to message queue from shop, get from message queue on other side; requires message queue service installation and infrastructure support.

Cloud services, like IronMQ, do infrastructure support, provide simple libraries for many languages, have great web-based UI, etc.

The same for IronWorker, etc. systems for asynchronous processing. For example, if I have small app, I can do such jobs right on my server. It only requires to use specialized library. In case of medium app, I can setup server and install software for asynchronous tasks processing. If I have huge app with a lot of tasks, it requires to support scalable infrastructure, tasks load balancer, etc.

Companies which provide cloud services for developers are programmed infrastructure controller layer over services like AWS. They provide easy API to their services, support infrastructure and provide a lot of client libraries for different languages.