|
/usr/local/perl/lib/site_perl/5.6.1/Schedule/Parallel.pm |
Schedule::Parallel
use Schedule::Parallel;
@unused_portion = runqueue jobcount, closure, [closure ...];
Fork with a bit of boilerplate and a maximum number of jobs to run at one time (jobcount). The queue (whatever's left on @_ after shifting off the count) is run in parallel by forking perl and exiting the sub-process with the closure's status.
The caller gets back the unexecuted portion of the queue. In a scalar context this will return false if the entire execution succeeded; in an array context it returns the unused portion for your money bac... er, in order to simplify re-execution where the calling code can fix the problems (e.g., if the closures store recovery information).
fork + exit semantics require that code called from the closures exits zero if it succeeds (i.e., shell-like returns). Any non-zero exit from a forked job will abort further processing.
For debugging: if jobcount is zero then the que is run without forking -- basically via $_->() for @_ on the whole que. This saves dealing with fork issues in devlopment.
Running N jobs in parallel makes the assumption that jobs will consume roughly constant system resource during execution. If this is not true it may be useful to submit a large que in sections with some monitoring to adjust the jobcount parameter as pieces of the queue are completed. Examples would be raising the value of jobcount for long-running, mostly-blocked queues (e.g., web searches) or reducing it to avoid bombarding the network if return times are faster.
Steven Lembark Workhorse Computing, LLC
(C) 2001-2004 Workhorse Computing, LLC
This code is released under the same terms as Perl istelf. Please see the Perl-5.6.1 distribution (or later) for a full description.
In any case, this code is release as-is, with no implied warranty of fitness for a particular purpose or warranty of merchantability.
perl(1)
|
/usr/local/perl/lib/site_perl/5.6.1/Schedule/Parallel.pm |