Laravel Queues - Passing Data to the Queue

Question

I have an array containing about ~8,000 stock tickers that I'm trying to queue up; the queue is meant to receive the array of stock tickers ($symbols[]), and then pass each one to a worker / consumer (whichever jargon you prefer).

Here's what my QueueController current looks like:

Class QueueController extends \BaseController {
    public function stocks()
    {
        $symbols = $this->select_symbols();
        Queue::push('StockQueue', array('symbols' => $symbols));
    }
    ...
}

From my QueueController, I'm calling a method to retrieve the list of stock symbols and passing it to the StockQueue Class as the $data.

public function fire($job, $data)
    {
        $symbols = $data; // print_r shows all symbols...

        // Get Quote Data for Symbol
        $quote = $this->yql_get_quote($symbol);

        // Get Key Stats for Symbol
        $keystats = $this->yql_get_keystats($symbol);

        // Merge Quote and Keystats into an Array
        $array[] = $quote;
        $array[] = $keystats;

        // Save Data to DB
        $this->yql_save_results($array, $symbol);

        $job->delete();
    } 

This is not what I'm trying to achieve though; what I need to do is pass in each symbol, one by one, to the StockQueue Class, and have it process it as a task.

If I were to wrap the StockQueue->stocks() method in a while loop, it would try and pass all ~8,000 in (from what I understand) immediately to the queue. Would this be detrimental or is this the best way to do it? I haven't been able to find a lot of examples for PHP-based RPC Message Queuing online, so I'm just as curious about the best practices as I am on the correct process.

With that being said, how can I fire up multiple workers for this queue? Say, I want 5 workers (depending on how many resources each one takes; I'll figure that out) to process these tasks in order to reduce the processing time by ~4/5ths. How would I do that?

Would I just launch php artisan queue:listen five times?

And, for clarity, I'm using beanstalkd and supervisord to do the message queue / monitoring.

I look forward to your advice and insight.

Answer 1

Yep, just run more workers. Beanstalkd can hold a number of connections open from lots of workers and make sure they all get different jobs. Just make sure that the job completes successfully (if not, deal with it appropriately - or at least bury it to look at later) and give it enough to complete, with some to spare in the TTR (Time To Run) setting.

As for how to run more jobs - yes, just increase the number of jobs available in Supervisord (numprocs=5 in the [program:NAME] section) and have them start. I tended to have another (larger) pool of the same jobs, that don't start automatically, so I could start a couple more manually through the Supervisord control, as required.