Scala dependency on Spark installation

Question

I'm just starting out with Spark, so I downloaded the binaries for Hadoop 1 (HDP1, CDH3)from here and extracted it to the Ubuntu virtual machine. Without installing Scala, I was able to run the examples in the Quick Start Guide from the Spark interactive shell.

• Is Spark enabled with Scala? If so, where are the libraries / binaries?
• To run Spark in other (distributed) modes, do I need to install Scala on all nodes?

As a note, I noticed that Spark has one of the best open source project documentation.

Answer 1

Is Spark enabled with Scala? If so, where are the libraries / binaries?

project/. :

$ ls project/
build.properties  plugins.sbt  project  SparkBuild.scala  target

sbt/sbt assembly, Scala . target/, :

$ ls target/
scala-2.9.2  streams

, Scala 2.9.2.

Spark () Scala ?

. ,

, , . jar ( "uber" jar), . sbt Maven . Spark ; , . , SparkContext, . SparkContext.

Answer 2

-

.

/SPARK_HOME///scala -2.9.3/ assembly_2.9.3-0.8.0--hadoop1.0.4.jar

scala + spark.

, CLASSPAH

: run spark-shell > http://: 4040 > a > Classpath

, scala , CLASSAPATH .

note: , , . :)

Answer 3

Scala . make-distribution.sh Scala . Scala, REPL, Scala jar.

, :

# The distribution contains fat (assembly) jars that include the Scala library,
# so it is completely self contained.
# It does not contain source or *.class files.

, Scala , make-distribution.sh.

Answer 4

Starting with spark 1.1, no SparkBuild.scala. You want to make changes to pom.xml and create using Maven