All geek questions in one place

Filing a spark application as a job for yarn from the context of Eclipse and Spark

I can already submit localspark jobs (written in Scala) from my Eclipse IDE. However, I would like to change my Spark context (inside my application) so that when the application starts (inside Eclipse) the task will be sent to my remote cluster, using Yarn as a resource manager.

Using spark-submit, I can successfully send the task to the cluster as: spark-submit --class <main class> --master yarn-cluster <jar>

I want to achieve the same result inside the IDE. My sbt config (application root directory) looks like this: libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" libraryDependencies += "org.apache.spark" %% "spark-yarn" % "1.5.1" % "provided" In my application: val conf = new SparkConf().setAppName("xxx").setMaster("yarn-cluster") However, I get the following errors:

Detected yarn-cluster mode, but isn't running on a cluster. Deployment to YARN is not supported directly by SparkContext. Please use spark-submit.

+4
source share
1 answer

1) , yarn-cluster Eclipse, spark-client.

new SparkConf().setAppName("test-app").setMaster("yarn-client");

Cloudera, , , "" .

2) , . , , ( - , src/java ):

  • core-site.xml
  • HDFS-site.xml
  • site.xml

, core-site.xml , , , . , fs.defaultFS , Spark , , ( ), HDFS.

+1

Source: https://habr.com/ru/post/1617726/

More articles:


All Articles