Run Spark-Submit on YARN, but imbalance (only 1 node works)

Question

I am trying to run Spark Apps in YARN-CLUSTER (2 nodes) , but it seems that these 2 nodes are an imbalance, because only 1 node works, but the other does not.

My Script:

spark-submit --class org.apache.spark.examples.SparkPi 
--master yarn-cluster --deploy-mode cluster --num-executors 2 
--driver-memory 1G 
--executor-memory 1G 
--executor-cores 2 spark-examples-1.6.1-hadoop2.6.0.jar 1000

I see that one of my node is working, but the other is not, so this is an imbalance:

Note: on the left namenode, and datanodeon the right ...

Any idea?

Answer 1

, . spark-submit

- conf "spark.locality.wait.node = 0"

.

Answer 2

yarn-cluster, Spark- ApplicationMaster

yarn-client, . , ,

spark-submit --verbose --class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode client \
--num-executors 2 \
--driver-memory 1G \
--executor-memory 1G \
--executor-cores 2 spark-examples-1.6.1-hadoop2.6.0.jar 10

Answer 3

, node SPARK

Spark UI ,

- TAB Spark UI