All geek questions in one place

Spark on yarn, connection to ResourceManager on / 0.0.0.0: 8032

I wrote a spark program on my developing machine, which is a mac. The hadoop version is 2.6, the spark version is 1.6.2. The hadoop clan has 3 nodes, of course, everything in the Linux machine. I run the spark program in IDE ID in spark offline mode, it works successfully. But now I change it to the yarn-client mode, it does not work successfully and gives a message as follows:

... 2017-02-23 11:01:33,725-[HL] INFO main org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at /0.0.0.0:8032 2017-02-23 11:01:34,839-[HL] INFO main org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2017-02-23 11:01:35,842-[HL] INFO main org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2017-02-23 11:01:36,847-[HL] INFO main org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2017-02-23 11:01:37,854-[HL] INFO main org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) ...

I have already added the appropriate configuration files to the project resource directory. If I create a jar package and use spark-submit to run this program, that will be fine. Now I want to run this program in the IDE as a "yarn-client" mode, as well as offline. How can I fix this problem? Thanks.

+5
source share
2 answers

Ensure that YARN configurations are available for use by Spark when operating in yarn mode. Add these core-site.xml , hdfs-site.xml and yarn-site.xml to the spark's conf directory.
Also make sure yarn-site.xml contains the address of the resource manager

 <property> <name>yarn.resourcemanager.address</name> <value>resource_manager_ip:8032</value> </property>
+2
source

Set your conf object this way, its working for me:

 conf = new SparkConf().setAppName(setup.getAppname).setMaster("yarn") .set("spark.hadoop.yarn.resourcemanager.hostname", "resourcemanager.fqdn") .set("spark.hadoop.yarn.resourcemanager.address", "resourcemanager.fqdn:8032")`

Font: hortonworks.com

0
source

Source: https://habr.com/ru/post/1264609/

More articles:


All Articles