All geek questions in one place

Python multithreading max_workers

According to ThreadPoolExecutor documentation

If max_workers is None or not specified, by default the number of processors on the machine will be indicated.

If I have not set a value like this

 ThreadPoolExecutor(max_workers=None)

Is this bad for performance if my value is very low (2)? Will python already allocate all processor processes for the value None vs allocate only 2 for the value with number?

+1
source share
1 answer

To begin with, you are referring to the wrong part of the documentation for your link, namely to processes, not to threads. one for concurrent.futures.ThreadPoolExecutor states:

Changed in version 3.5: If max_workers is None or not specified, by default the number of processors on the machine multiplied by 5 will be indicated, assuming that ThreadPoolExecutor is often used to overlap I / O instead of the central processor, the work and the number of workers should be higher than Number of employees for ProcessPoolExecutor.

Since you are using threads, not processes, it is assumed that your application is related to IO, not CPU binding, and that you use this for concurrency, not parallelism . The more threads you use, the higher concurrency you will reach (to the point), but the fewer CPU cycles you will get (as there will be context switches). You need to tune the application to typical workloads to see what works best for you. There is no universally optimal solution for this.

+1
source

Source: https://habr.com/ru/post/1275726/

More articles:


All Articles