/usr/local/perl/lib/site_perl/5.6.1/Schedule/Parallel/Thread.pm


NAME

Schedule::Parallel


SYNOPSIS

  use Schedule::Parallel;
  @unused_portion =  
          runqueue( jobcount, closure, [closure ...] );


DESCRIPTION

Fork with a bit of boilerplate and a maximum number of jobs to run at one time (jobcount). The queue (whatever's left on @_ after shifting off the count) is run in parallel by forking perl and exiting the sub-process with the closure's status.

The caller gets back the unexecuted portion of the queue. In a scalar context this will return false if the entire execution succeeded; in an array context it returns the unused portion for your money bac... er, in order to simplify re-execution where the calling code can fix the problems (e.g., if the closures store recovery information).

fork + exit semantics require that code called from the closures exits zero if it succeeds (i.e., shell-like returns). Any non-zero exit from a forked job will abort further processing.

Notes

Running N jobs in parallel makes the assumption that jobs will consume roughly constant system resource during execution. If this is not true it may be useful to submit a large que in sections with some monitoring to adjust the jobcount parameter as pieces of the queue are sumitted. Examples would be raising the value of jobcount for long-running, mostly-blocked queues (e.g., web searches) or reducing it to avoid bombarding the network if return times are faster.

The leading newlines may look a bit odd but help separate status notices from any sub-proc output that they get buried in. Ditto the leading ``$$'' on all of the messages.

One minor modification to the arguments improves the logging. Instead of passing in an array of closures, use an array of arrays, with the title and a closure. The title is printed, the closure is executed. In cases where the called subroutine doesn't give sufficient logging information this might help. The required change is simply:

        my $item = shift;
        my ( $title, $morsel ) = @$item;

and print $title in the status messages:

        "\n$$: Forked $forkfull running $title\n";

and

        "\n$$: Running $title";

Should provide sufficient logging information for most purposes.


AUTHOR

Steven Lembark Knightsbridge Solutions, LLC

slembark@knightsbridge.com


Copyright

(C) 2001-2002 Steven Lembark, Knightsbridge Solutions

This code is released under the same terms as Perl istelf. Please see the Perl-5.6.1 distribution (or later) for a full description.

In any case, this code is release as-is, with no implied warranty of fitness for a particular purpose or warranty of merchantability.


SEE ALSO

perl(1).

 /usr/local/perl/lib/site_perl/5.6.1/Schedule/Parallel/Thread.pm