What's the best way of executing tasks in parallel in Ksh and Perl?

Question

I have this large C++ project that I need to build on a platform that does not have a parallel make (like make -j on Linux). The server has 6 CPU's and I want to do a parallel build manually.

I can generate a task list like this for about 300 object files. I use the Makefile for the dependency checks and incremental build:

make -f Makefile obj1.o

make -f Makefile obj2.o

make -f Makefile obj3.o ...

How would I execute these tasks in parallel with no more then 6 tasks running at a time using Ksh and Perl? (Java or Python are not available :-( )

Answer 1

In Perl the you should look at Parallel::ForkManager. You could do something like this:

my @make_obj = qw(
  obj1.o
  obj2.o
  obj3.o
  ...
);

my $fm = $pm = new Parallel::ForkManager(6);
foreach my $obj (@make_obj) {
  $fm->start and next;
  system("make -f Makefile $make_obj");
  $fm->finish();
}

Answer 2

An alternative to forking is to run each make in its own thread.

use threads;

my $max_threads = 5;

my @targets = qw(obj1.o obj2.o obj3.o ...);
while(@targets) {
    my $num_threads = threads->list(threads::running);
    if( $num_threads < $max_threads ) {
        my $target = shift @targets;
        threads->create(sub { return system "make $target" });
    }
}

I am unfortunately hand waving around two bits. First is making the loop wait quiescently until a thread finishes. I believe this is accomplished using threads::shared's cond_wait() and a semaphore variable.

The second is getting the return value from make, so you know if something failed and to halt the build. To do that you'd have to join() each thread to get the result of system(), the process' exit code.

Sorry for the hasty reply. Hopefully the community will fill in the rest.

Answer 3

Does gnu make on HPUX not have the -j flag?

http://hpux.connect.org.uk/hppd/hpux/Gnu/make-3.81/

Answer 4

Using GNU Parallel you can write:

parallel -j6 make -f Makefile obj{}.o ::: {1..500}

10 second installation:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

Learn more: http://www.gnu.org/software/parallel/parallel_tutorial.html https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Answer 5

If the OS is properly handling inter-processor communication and scheduling, you should just be able to throw all the makes into backgrounded processes, as long as there are no interdependencies.

If they're all independent, I would issues a series of commands like this:

make -f Makefile obj1.o &
make -f Makefile obj2.o &
...
make -f Makefile objn.o &

With dependencies, string them with the double ampersand (&&) so that the dependent item doesn't start until its 'parent' has finished.

I realize that's not quite the solution you requested, but it's how I would attack the problem :)