@unused_portion = runqueue( jobcount, closure, [closure ...] );
Fork with a bit of boilerplate and a maximum number of jobs to run at one time (jobcount). The queue (whatever's left on @_ after shifting off the count) is run in parallel by forking perl and exiting the sub-process with the closure's status.
The caller gets back the unexecuted portion of the queue. In a scalar context this will return false if the entire execution succeeded; in an array context it returns the unused portion for your money bac... er, in order to simplify re-execution where the calling code can fix the problems (e.g., if the closures store recovery information).
fork + exit semantics require that code called from the closures exits zero if it succeeds (i.e., shell-like returns). Any non-zero exit from a forked job will abort further processing.
Running N jobs in parallel makes the assumption that jobs will consume roughly constant system resource during execution. If this is not true it may be useful to submit a large que in sections with some monitoring to adjust the jobcount parameter as pieces of the queue are sumitted. Examples would be raising the value of jobcount for long-running, mostly-blocked queues (e.g., web searches) or reducing it to avoid bombarding the network if return times are faster.
The leading newlines may look a bit odd but help separate status notices from any sub-proc output that they get buried in. Ditto the leading ``$$'' on all of the messages.
One minor modification to the arguments improves the logging. Instead of passing in an array of closures, use an array of arrays, with the title and a closure. The title is printed, the closure is executed. In cases where the called subroutine doesn't give sufficient logging information this might help. The required change is simply:
my $item = shift; my ( $title, $morsel ) = @$item;
and print $title in the status messages:
"\n$$: Forked $forkfull running $title\n";
"\n$$: Running $title";
Should provide sufficient logging information for most purposes.
Steven Lembark Knightsbridge Solutions, LLC
(C) 2001-2002 Steven Lembark, Knightsbridge Solutions
This code is released under the same terms as Perl istelf. Please see the Perl-5.6.1 distribution (or later) for a full description.
In any case, this code is release as-is, with no implied warranty of fitness for a particular purpose or warranty of merchantability.