How to submit a Scala task to Spark?

Question

I have a Pythons script that I was able to send Spark as follows:

/opt/spark/bin/spark-submit --master yarn-client test.py

Now I am trying to send the Scala program in the same way:

/opt/spark/bin/spark-submit --master yarn-client test.scala

As a result, you receive the following error message:

Error: Cannot load main class from JAR file:/home/myname/spark/test.scala
Run with --help for usage help or --verbose for debug output

The Scala program itself is a Hello World program:

object HelloWorld {
    def main(args: Array[String]): Unit = {
        println("Hello, world!")
    }
}

What am I doing wrong?

Answer 1

First you need to create a jar file. You cannot just send the Scala source. If in doubt, see Getting started with sbt .

After that, just add a parameter classpointing to HelloWorld. If there are no packages:

/opt/spark/bin/spark-submit --master yarn-client --class "HelloWorld" path_to.jar

Answer 2

, .

./bin/spark-submit \
  --class <main-class>
  --master <master-url> \
  --deploy-mode <deploy-mode> \
  --conf <key>=<value> \
  ... # other options
  <application-jar> \
  [application-arguments]

-

/opt/spark/bin/spark-submit \
  --class "HelloWorld" your_jar_with_scala_file \
  --master yarn-client

.