How does a Spark artist perform several tasks?

Question

For example, if the number of performers is 40, but the number of tasks is 80, this means that each performer will run two tasks in parallel. Also, my functions (the execution of which are executed) are not ordinary functions, but I call programs inside them. Thus, each task actually takes several minutes. So my question is, how does Spark handle it? Will these tasks be shared by the JVM artist? What about the number of cores, do you divide it between these two tasks? What to do if you do not want these two tasks to be performed simultaneously, but do them cyclically, i.e. Run the first task with all the kernels of the performers and only when to finish it, start the second task?

Answer 1

depends on how you allocate your resources, that is, the number of cores, cores per artist and allocated memory to the performers. it also depends on how you program to achieve maximum parallelism.

it also depends on how you code to achieve maximum parallelism. if there are two tasks, and they are independent of each other, they will work in parallel. if one task depends on the result of the previous task, they will be performed in series.

Yes, the number of cores can be divided into two tasks, making two executors and highlighting the available cores.

. , .