How does Amazon's SQS notify one of my “worker” servers whenever there is something in the queue?

Question

I'm following this tutorial: http://boto.s3.amazonaws.com/sqs_tut.html

When there's something in the queue, how do I assign one of my 20 workers to process it?

I'm using Python.

Answer 1

Unfortunately, SQS lacks some of the semantics we've often come to expect in queues. There's no notification or any sort of blocking "get" call.

Amazon's related SNS/Simple Notification Service may be useful to you in this effort. When you've added work to the queue, you can send out a notification to subscribed workers.

See also:

http://aws.amazon.com/sns/

Best practices for using Amazon SQS - Polling the queue

Answer 2

This is (now) possible with Long polling on a SQS queue.

http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/Query_QueryReceiveMessage.html

Long poll support (integer from 1 to 20) - the duration (in seconds) that the ReceiveMessage action call will wait until a message is in the queue to include in the response, as opposed to returning an empty response if a message is not yet available.

If you do not specify WaitTimeSeconds in the request, the queue attribute ReceiveMessageWaitTimeSeconds is used to determine how long to wait.

Type: Integer from 0 to 20 (seconds)

Default: The ReceiveMessageWaitTimeSeconds of the queue.

Answer 3

Actual if you dont require a low latency, you can try this:

Create an cloudwatch alarm on your queue, like messages visible or messages received > 0. As an action you will send a message to an sns topic, which then can send the message to your workers via an http/s endpoint.

normally this kind of approach is used for autoscaling.

Answer 4

Further to point out a problem with SQS - You must poll for new notifications, and there is no guarantee that on any particular poll you will receive an event that exists in the queue (this is due to the redundancy of their architecture). This means you need to consider the possibility that your polling didn't return a message that existed (which for me meant I needed to increase the polling rate).

All in all I found too many limitations with SQS (as I've found with some other AWS tools such as SimpleDB). But that's just my injected opinion.

Answer 5

There is now an JMS wrapper for SQS from Amazon that will let you create listeners that are automatically triggered when a new message is available.

http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/jmsclient.html#jmsclient-gsg