Python 3 multiprocessing.Process inside a class?

Question

I have a complex class A that calculates data (large matrix calculations) while consuming input from class B.

A itself uses several cores. However, when A needs the next piece of data, it waits a rather long time, since B works in the same main stream.

Since A mainly uses the GPU for computing, I would like B to collect data simultaneously on the CPU.

My last approach:

# every time *A* needs data def some_computation_method(self): data = B.get_data() # start computations with data 

... and B looks something like this:

 class B(object): def __init__(self, ...): ... self._queue = multiprocessing.Queue(10) loader = multiprocessing.Process(target=self._concurrent_loader) def _concurrent_loader(self): while True: if not self._queue.full(): # here: data loading from disk and pre-processing # that requires access to instance variables # like self.path, self.batch_size, ... self._queue.put(data_chunk) else: # don't eat CPU time if A is too busy to consume # the queue at the moment time.sleep(1) def get_data(self): return self._queue.get() 

Can this approach be considered a "python" solution?

Since I have little experience with the Python multiprocessing module, I created a simple / simplified approach. However, for me it looks like “hacks."

What would be the best solution for simultaneously loading Class B data from the disk and feeding it through a queue while the main thread does heavy calculations and consumes data from the queue from time to time?

Answer 1

While your solution is fine, especially for "small" projects, it has a threading flaw closely associated with class B Therefore, if you (for example) for some reason wanted to use B non-threaded manner, you're out of luck.

I will personally write the class in streaming safe mode, and then call it using streams from the outside:

 class B(object): def __init__(self): self._queue = multiprocessing.Queue(10) ... if __name__ == '__main__': b = B() loader = multiprocessing.Process(target=b._concurrent_loader) loader.start() 

This makes B more flexible, better separates dependencies, and easier to test. It also makes the code more readable by being explicit about creating a stream, compared to implicitly implying a class is created.