Asynchronously running perl's backticks

Question

Right now I have a perl script that at a certain point, gathers and then processes the output of several bash commands, right now here is how I've done it:

if ($condition) {
  @output = `$bashcommand`;
  @output1 = `$bashcommand1`;
  @output2 = `$bashcommand2`;
  @output3 = `$bashcommand3`;
}

The problem is, each of these commands take a fairly amount of time, therefore, I'd like to know if I could run them all at the same time.

Answer 1

This sounds like a good use case for Forks::Super::bg_qx.

use Forks::Super 'bg_qx';
$output = bg_qx $bashcommand;
$output1 = bg_qx $bashcommand1;
$output2 = bg_qx $bashcommand2;
$output3 = bg_qx $bashcommand3;

will run these four commands in the background. The variables used for return values ($output, $output1, etc.) are overloaded objects. Your program will retrieve the output from these commands (waiting for the commands to complete, if necessary) the next time those variables are referenced in the program.

... more stuff happens ...
# if $bashcommand is done, this next line will execute right away
# otherwise, it will wait until $bashcommand finishes ...
print "Output of first command was ", $output;

&do_something_with_command_output( $output1 );
@output2 = split /\n/, $output2;
...

Update 2012-03-01: v0.60 of Forks::Super has some new constructions that let you retrieve results in list context:

if ($condition) {
    tie @output, 'Forks::Super::bg_qx', $bashcommand;
    tie @output1, 'Forks::Super::bg_qx', $bashcommand1;
    tie @output2, 'Forks::Super::bg_qx', $bashcommand2;
    tie @output3, 'Forks::Super::bg_qx', $bashcommand3;
}
...

Answer 2

On a Unix system, you should be able to open multiple command pipes, then run a loop calling IO::Select to wait for any of them to be ready to read; keep reading and slurping their output (with sysread) until they all reach end of file.

Unfortunately, apparently Win32 emulation of Unix select can't handle file I/O, so to pull it off on Windows you'll also have to add a layer of socket I/O, for which select works, see perlmonks.

Answer 3

You can, but not using backticks.

Instead you'll need to open real file handles for them using open(handle, "$bashcommand|"); and then do a proper select call to determine which one has new output that is ready for you. It will take a lot more than the 6 lines you have above, but you will be able to run them all at the same time.

There are some classes in CPAN that likely already manage some of the complexity for you.

Answer 4

You should refer to Perl FAQ.

Proc::Background looks promising.