All geek questions in one place

What happens if I try to use more cores than mine?

In my intrinsically safe I can set the number of cores to use, I have 4 physical, 8 logical on my laptop, what does the spark do if I indicate a number that is impossible on the machine, for example, 100 cores?

+5
source share
2 answers

The number of cores does not describe the physical cores, but the number of working threads. This means that nothing strange happens if the number is higher than the number of available cores.

Depending on your setup, this may actually be the preferred configuration with a value of about two times the available cores available, which are usually recommended. Obviously, if the number is high, your application will spend more time switching between threads than the actual processing.

+7
source

It depends on your cluster manager. I assume that you are asking about the local[n] runtime.

If so, then the driver and one and only one executor is the same JVM with the number of threads n .

DAGScheduler - Spark's execution scheduler will use n threads to schedule as many tasks as you have been told.

If you have more tasks, i.e. threads than cores, your OS will have to deal with more threads than cores, and plan them accordingly.

0
source

Source: https://habr.com/ru/post/1241089/

More articles:


All Articles