Spawning child workers from a master worker using Iron.io's IronWorker service

Question

Currently, I have a PHP script (e.g. masterProcessor.php) that my server executes at regular intervals using cron. This script has a static list of about 80 URLs it must fetch and process. Since processing each URL takes a few minutes, to save time the script divides the 80 URLs up into about 10 sets of 8, and fires off a second PHP script (using exec("childProcess.php")) for each set of URLs, and that child PHP script actually processes each of the 8 or so URLs it is handed from the master script.

My aim is to accomplish this using Iron.io's IronWorker service, but I'm still a bit confused on how to go about this. So much of their docs are in Ruby, which I don't know, and the few PHP examples they have show code but not how to actually set this up.

Here is how I think this would work, so please let me know if I'm right or wrong here:

1. Create a "master" worker and a "child" worker.
2. The master worker would fire off child tasks, sending each task a payload containing the URLs to process.
3. The child worker would process the URLs given to it in the payload.
4. Finally, I'd need to schedule the master worker to run at regular intervals, much as I do via cron with my PHP implementation.

Do I have that right? Is it even possible for a worker to fire off other, multiple workers/tasks? If so, how is that accomplished using the PHP IronWorker lib?

Any guidance or tips, resources, etc. would be much appreciated. I apologize for my ignorance, but I've been reading and researching and I've tried experimenting locally but I can't even get a single worker to run locally on Windows (it says it got executed, but no logs are printed out?).

Answer 1

I'd suggest to try https://github.com/iron-io/iron_combine - it's small helper framework with master script already implemented. You'll just need to implement slave and push 1 message for each url to mq.

In any case - your approach is correct, if you don't want to use iron_combine for whatever reason, just do it as you described in question. If you'll face any problems, iron.io got nice support channel http://www.hipchat.com/gym1ayjWj

Answer 2

Easy, if you need child workers

First of all, you need

• client library https://github.com/iron-io/iron_worker_php/blob/master/iron_worker.phar and
• credentials data - you could download iron.json directly from HUD

Now:

1) merge these two files into master worker:

file 'iron_worker.phar'
file 'iron.json'

2) and use it as usual:

<?php
require_once "phar://iron_worker.phar";

$worker = new IronWorker();
$worker->postTask('SlaveWorkerName', $payload);

3) don't forget to upload both master and slave worker

and 4) scheduling - could be done once using the UI (http://hud.iron.io -> project -> worker -> scheduled tasks -> add) or using $worker->postSchedule() code - see https://github.com/iron-io/iron_worker_php/blob/master/IronWorker.class.php#L631

Answer 3

I am not for sure how PHP IronWorker occurs but you can use the pcntl_fork function to make child processes. Then your script would have to keep track of each one of these processes. Also these functions only can be used in cli and not through a web interface. http://us2.php.net/pcntl_fork

Here is a tutorial on multiprocessing.

http://www.tuxradar.com/practicalphp/16/1/3

What about breaking the script up in a couple of different files. Each file does 10, 20, or etc. about of the urls. Might be easier then trying to learn how processes management works.