A queuing system which supports job batching (e.g. several jobs for 1 worker at once)

Question

I'm looking for a queuing system that could support the following scenario:

• A client adds a job - to check how many Facebook likes a particular url (URL1) has;
• A client adds another job - to check the same information for URL2;

• [....]

• A worker picks up anything from 1 to 50 jobs (urls) from the queue (e.g., if there's only 5 - it picks up 5, if there's 60 - picks up 50, leaving others for another worker), and issues a request against Facebook API (which allows multiple urls per request). If it succeeds, all jobs are taken out of the queue, if it fails - all of them stay.

I'm working with PHP and I've looked into Gearman, Beanstalkd, but did not find any similar functionality. Is there any (free) queuing system that would support such a "batch-dequeuing"?

Or, maybe, anybody could suggest an alternative way of dealing with such an issue? I've considered keeping a list of "to check" urls outside the queuing system and then adding them in bundles of max N items with a cron job that runs every X period. But that's kind of building your own queue, which defeats the whole purpose, doesn't it?

Answer 1

I've used Beanstalkd to fetch 100 twitter names at a time, and then calling an API with them all. When I was done, I deleted them - but I could have elected to not delete some (or all) if I wished.

It was a simple loop to reserve the initial 100 (one at a time), and I put the results (the job ID and the data returned) into an array. When I was done dealing with the payload (in this instance, a twitter screen-name), I went through deleting them - but I could have easily have released them back into the queue.

Answer 2

Perhaps you could take inspiration from MediaWiki's job queue system. Not very complicated, but it does have some issues that you may run into if you decide to roll your own.

The DB tables used for this are defined here.