Python Multiprocessing - just not getting it

Question

I spent some time trying to understand multiprocessing, although its subtle points deviate from my unprepared mind. I managed to get the pool to return a prime integer , but if the function does not just return the result, like all the examples that I can find (even in the documentation , this is some kind of obscure example that I can not understand.

Here is an example I'm trying to get. BUT, I can't get it to work properly, and I'm sure there is a simple reason. I may have to use a queue or shared memory or a dispatcher, but as many times as I read the documentation, I seem to be unable to get around my brain around what it really means and what it does. All I could understand so far is the pool function.

In addition, I use the class as I need to avoid using global variables, as in the answer to this question . In the end, it will run on multiple cores, so I guess I need

import random class thisClass: def __init__(self): self.i = 0 def countSixes(myClassObject): newNum = random.randrange(0,10) #print(newNum) #this proves the function is being run if enabled if newNum == 6: myClassObject.i += 1 if __name__ == '__main__': import multiprocessing pool = multiprocessing.Pool(1) #use one core for now counter = thisClass() myList = [] [myList.append(x) for x in range(1000)] #it must be (args,) instead of just i, apparently async_results = [pool.apply_async(countSixes, (counter,)) for i in myList] for x in async_results: x.get(timeout=1) print(counter.i) 

Can someone explain to stupidly stupid what needs to be done so that I can finally understand what I am missing and what he is doing?

Answer 1

It took me a while to figure out what you want. The problem is how multiprocessing works. Basically, you need to write your program in a functional style, instead of relying on the side effects as it is now.

Right now, you are sending objects to your pool that you need to change, and do not return anything from countSixes . This will not work with multiprocessing, because in order to get around the GIL , multiprocessing creates a copy of counter and sends it to a completely new translator. Therefore, when you increase i , you actually increase the copy of i , and then because you are not returning anything, you are dropping it!

To do something useful, you need to return something from countSixes . Here is a simplified version of your code that does something similar to what you want. I left the argument, just to show what you should do, but actually it can be done using the zero-arg function.

 import random def countSixes(start): newNum = random.randrange(0,10) if newNum == 6: return start + 1 else: return start if __name__ == '__main__': import multiprocessing pool = multiprocessing.Pool(1) #use one core for now start = 0 async_results = [pool.apply_async(countSixes, (start,)) for i in range(1000)] print(sum(r.get() for r in async_results))