How to convert a nested loop of a loop to a multi-threaded program in perl

Question

I need help converting a nested loop to a multithreading program in Perl, for example.

for ( my $i=0; $i<100; $i++) { for ( my $j=0; $j<100; $j++ ) { for ( my $k=0; $k<100; $k++ ) { #do something .... } } } 

Is there a way I can break the first loop as shown below and run them in parallel

 #Job1: for ( my $i=0; $i < 40; $i++) { for( my $j=0; $j < 100; $j++) { for( my $k=0; $k < 100; $k++) { #do something .... } } } #Job2: for ( my $i=40; $i < 80; $i++) { for( my $j=0; $j<100; $j++) { for( my $k=0; $k<100; $k++) { #do something .... } } } #Job3 for ( my $i=80; $i < 100; $i++) { for( my $j=0; $j < 100; $j++) { for( my $k=0; $k < 100; $k++) { #do something .... } } } 

How can I run each program in parallel, and then exit the main program only after completing all the routines Job1, Job2 and job3.

Answer 1

I offer a link to a similar answer that I used before - the key question is - are your works completely untied? For instance. Do I need to move between them?

If so, use Parallel::ForkManager , it looks a bit like this:

 use Parallel::ForkManager; my $fork_manager = Parallel::ForkManager -> new ( 10 ); #10 in parallel for ( my $i=0;$i<100;$i++) { #in parallel: $fork_manager -> start and next; for ( my $j=0; $j < 100; $j++) { for ( my $k=0; $k < 100; $k++) { #do something .... } } $fork_manager -> finish; } $fork_manager -> wait_all_children(); 

This will be for each iteration of $i fork code and run in parallel - and ForkManager will close concurrency by 10.

This number should be roughly comparable to the limiting factor in your parallelism - if it's the CPU, then the number of processors, but keep in mind that you are often more limited by disk IO.

Key caveats for doing parallelism:

• You cannot guarantee a run without mess. It is possible that the cycle $i==1 ends after the cycle $i==2 . Or earlier. Or whatever.

• If you transfer information between your cycles, parallel efficiency loses efficiency - because the sender and receiver need to synchronize each of them. This is even worse if you need to synchronize the entire batch, so try not to do this more than necessary. (for example, when possible, leave it to the end and compare the results).

• This doubles for branched code - these are separate processes, so you really need to try and pass things back and forth.

• Because of this first point, you can get very bad errors from parallel code. Individual lines of code can occur in any order, so very strange things can happen. Each process will alternate, but several may alternate. Something harmless like open ( my $file, ">>", $output_filename ); maybe move you.

• The layout is quite limited in its ability to share data between forks. If you need to do most of this, consider flowing.

Threading is an alternative concurrency model that can be useful in certain circumstances. I usually tend to fork ing, as a rule, "better", but in those places where I want to do the honest part of interprocess communication, I will tend to look more at threads . Perl daemonize with child daemons