How to reuse a process pool for parallel programming in Python 3

Question

I am new to parrallel programming. My task is to analyze hundreds of data files. Each of these data is about 300 MB and can be cut into numerous fragments. My computer is a 4-core PC. And I want to get the result of each information as soon as possible.
The analysis of each data file consists of two procedures. Read the data in memory first, and then cut it into slices, which is intense work. Then do a lot of calculations for the fragments of this file, which is intensively associated with the processor.
Therefore, my strategy groups these files in a group of 4. For each group of these files, first read all the data from 4 files into memory with 4 processes in 4 cores. The code is similar

with Pool(processes=4) as pool:
    data_list = pool.map(read_and_slice, files)  # len(files)==4

Then, for each datain, data_listperform calculations with 4 processes.

for data in data_list:  # I want to get the result of each data asap
    with Pool(processes=4) as pool:
        result_list = pool.map(compute, data.slices)  # anaylyze each slice of data
    analyze(result_list)  # analyze the results of previous procedure, for example, get the average.

.
, . ? ? , ?

!

Answer 1

with Pool for...

p = Pool()
for data in data_list:
  result_list = pool.map(compute, data.slices)
  analyze(result_list)
p.join()
p.close()

python 2 3.

( ) pathos, from pathos.pools import ProcessPool as Pool , , Pool. , pathos Pool, Pool, , . pool.terminate(), .

>>> from pathos.pools import ProcessPool as Pool
>>> pool = Pool()
>>> data_list = [range(4), range(4,8), range(8,12), range(12,16)]
>>> squared = lambda x:x**2
>>> mean = lambda x: sum(x)/len(x)
>>> for data in data_list:
...   result = pool.map(squared, data)
...   print mean(result)
... 
3
31
91
183

, pathos , for (amap from pathos)... (imap_unordered multiprocessing uimap pathos). . : https://stackoverflow.com/questions/28203774/how-to-do-hierarchical-parallelism-in-ipython-parallel : fooobar.com/questions/1622410/...

. pathos python2. ( ) python3.