In fact, this answer is not as simple as one would expect.
I will make a lot of assumptions, first I use sbt , secondly, you work on a Linux-based computer, the third is that you have two classes in your project, say RunMe and Globals , and the last assumption will be that You want to adjust the parameters inside the program. So, somewhere in your executable code, you should have something like this:
object RunMe { def main(args: Array[String]) { val conf = new SparkConf() .setMaster("mesos://master:5050") //If you use Mesos, and if your network resolves the hostname master to its IP. .setAppName("my-app") .set("spark.executor.memory", "10g") val sc = new SparkContext(conf) val sqlContext = new SQLContext() //your code comes here } }
You must follow these steps:
Compile the project in its root using:
$ sbt assembly
Submit the task to the node wizard, this is the interesting part (assuming that you have the following structure in your target/scala/ project, and inside you have a .jar file that corresponds to the compiled project)
$ spark-submit --class RunMe target/scala/app.jar
Note that since I suggested that the project has two or more classes, you will need to determine which class you want to run. In addition, I am sure that both approaches, for Yarn and Mesos , are very similar.