Identifying and differentiating among workers when using Backburner for Beanstalkd

Question

I'm trying to use Backburner (a Ruby client for Beanstalkd) in my application to create, say, 50 workers all watching the same tube for incoming jobs. The problem is, each of those 50 workers, when performing a job, does some heavy file manipulation, etc., and each of them needs to perform some operations on a uniquely-named numbered directory. For example, worker 7 needs to work with a directory named dir7.

Initially I thought I could make a ThreadsOnFork worker with 50 threads, and I'd be able to access a number associated with each thread from the worker's perform() method, but I haven't been able to do this yet.

How do I handle this situation using Backburner? Or, if it's not possible with Backburner, using some other Ruby beanstalkd client? I hope I'm getting my question across clearly enough. Thanks.

Answer 1

I solved the issue, the crude way: wrote a shell script that starts a worker by passing its designated number to it via environment variables. Below is a snippet, if anyone's looking for a similar solution:

#!/bin/bash
for i in $(seq 1 "$1"); do
    env WORKER_NUM=$i rake QUEUES=queue1 backburner:work > logs/backburner.log 2>&1 &
done

The rake task becomes available by adding require 'backburner/tasks' to the Rakefile, as explained in the Backburner documentation.