Parallel computation of comb_with_replacement using multiprocessing

Question

I try to get every possible combination with a replacement and do some calculations with each of them. I am using the following code:

from itertools import combination_with_replacement

for seq in combination_with_replacement('ABCDE', 500):
    # some calculation

How can I parallelize this computation using multiprocessing?

Answer 1

You can use the standard library concurrent.futures.

from concurrent.futures import ProcessPoolExecutor
from itertools import combinations_with_replacement

def processing(combination):
    print(combination)
    # Compute interesting stuff


if __name__ == '__main__':
    executor = ProcessPoolExecutor(max_workers=8)
    result = executor.map(processing, combinations_with_replacement('ABCDE', 25))
    for r in result:
         # do stuff ...

A bit more explanation:

• processes. threads, python , , .
• . , executor.map.. , for.
• processing if __name__ == '__main__': . pickle . , , .

multiprocessing.Pool, , .

, 500 5 ABCDE . 5**500 > 1e350. , max_workers, 8, ~ 1e349, ~ 1e335 , 1 .