Why does Spark send a spark-submit script to ignore `-num-executors`?

Question

We have Spark 1.0.0 running under YARN, but --num-executorsdoes not seem to increase the number of artists or nodes used. I say that I need 8, but usually I get from 3 to 5. There are no errors in the output, what I expect if there are nodes where down and with which it would be impossible to talk.

NOTE. If you do NOT work under YARN, then num-executors will be ignored, for example. spark offline mode. See Accepted Answer for a solution and comments.

UPDATE: if I ask for X resources, I want to use X resources, and if I can’t, I want to be queued or given some kind of error message. This is because my work will fail if I do not get X resources - I know how many resources I need before my work goes down. I don’t want to implement some extra layer on top of my work in order to check how many artists and nodes I need to give, so that he can gracefully kill the work before it explodes on its own. So the second part of the question is “1”, is there any way to tell YARN / Spark to fail if I can't get the artists I want? 2) force stop YARN from placing more than one artist on the same node "

(In 0.9.0, this was not a problem, N nodes meant that N workers and tasks would simply be queues)

Answer 1

So yes, the reason that was --num-executorsnot respected in my original situation (i.e. under YARN) was due to some kind of erroneous behavior when it does not give you all the performers if it leads you to max core / memory .

One way (a) to protect against this (and thus answer my second question) and (b) force a number of performers to start the intrinsically safe mode (and thus referring to the note) is to transfer all the performer kernels and artist kernels configs to spark-submit and automatically calculate the total artist kernels using a script:

total_executor_cores=`expr ${num_executors} \* ${executor_cores}`

, , "", .

YARN, , , .

. .

Answer 2

, SparkSubmit --num-executors ( SPARK_EXECUTOR_INSTANCES spark.executor.instances). , - ( / ).

1) YARN/Spark , , ?

, . YARN . YARN , . , Driver, ApplicationMaster.

2) YARN node "

/ , .