Spark streaming elasticsearch dependencies

Question

Im trying to integrate Spark and Elasticsearch into scala as described in the Elasticsearch Guide

I am having problems with dependencies when compiling:

[trace] Stack trace suppressed: run last *:update for the full output. [error] (*:update) sbt.ResolveException: unresolved dependency: cascading#ing-local;2.5.6: not found [error] unresolved dependency: clj-time#clj-time;0.4.1: not found [error] unresolved dependency: compojure#compojure;1.1.3: not found [error] unresolved dependency: hiccup#hiccup;0.3.6: not found [error] unresolved dependency: ring#ring-devel;0.3.11: not found [error] unresolved dependency: ring#ring-jetty-adapter;0.3.11: not found [error] unresolved dependency: com.twitter#carbonite;1.4.0: not found [error] unresolved dependency: cascading#cascading-hadoop;2.5.6: not found [error] Total time: 86 s, completed 19 nov. 2014 08:42:58 

My build.sbt file is as follows

 name := "twitter-sparkstreaming-elasticsearch" version := "0.0.1" scalaVersion := "2.10.4" // additional libraries libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % "1.1.0", "org.apache.spark" %% "spark-streaming" % "1.1.0", "org.apache.spark" %% "spark-streaming-twitter" % "1.1.0", "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0" ) 

Help? thanks.

Answer 1

Cascading and its dependencies are not available in central Maven, but in their own repo (which es-hadoop cannot specify through its pom).

I solved the problem using elasticsearch-spark_2.10

http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/master/install.html

Answer 2

Sbt cannot resolve some dependencies as they are not part of the Maven repository. However, you can find them on clojars and conjars . You need to add the following lines so sbt can resolve them:

 resolvers += "clojars" at "https://clojars.org/repo" resolvers += "conjars" at "http://conjars.org/repo" 

Also, there is no elasticsearch-hadoop dependency "2.1.0" (yet?), You should use "2.1.0.Beta4" (or any other latest version when you read this)

Your sbt file should look like this:

 name := "twitter-sparkstreaming-elasticsearch" version := "0.0.1" scalaVersion := "2.10.4" libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % "1.1.0", "org.apache.spark" %% "spark-streaming" % "1.1.0", "org.apache.spark" %% "spark-streaming-twitter" % "1.1.0", "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.Beta4" ) resolvers += "clojars" at "https://clojars.org/repo" resolvers += "conjars" at "http://conjars.org/repo" 

This has been tested (with spark core 1.3.1 and no sparking, but it should work for you). Hope this helps.

Answer 3

This is because cascading and its dependencies are not in Maven. You must add recognizers to get them

Add this line to your build.sbt file

 resolvers += "conjars.org" at "http://conjars.org/repo" 

Your build.sbt file should look like this:

 name := "twitter-sparkstreaming-elasticsearch" version := "0.0.1" scalaVersion := "2.10.4" // additional libraries libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % "1.1.0", "org.apache.spark" %% "spark-streaming" % "1.1.0", "org.apache.spark" %% "spark-streaming-twitter" % "1.1.0", "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0" ) resolvers += "conjars.org" at "http://conjars.org/repo" 

NOTE. This issue has been raised and closed here https://github.com/elasticsearch/elasticsearch-hadoop/issues/304 with the same solution as above

Answer 4

"org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0" is not yet available. You can use "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.Beta2"

Check here:

• http://www.mvnrepository.com/artifact/org.elasticsearch/elasticsearch-hadoop/2.1.0.Beta2

EDIT1

Even after using "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.Beta2" I also get the same error:

 [info] Resolving org.fusesource.jansi#jansi;1.4 ... [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] :: UNRESOLVED DEPENDENCIES :: [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] :: cascading#cascading-local;2.5.6: not found [warn] :: clj-time#clj-time;0.4.1: not found [warn] :: compojure#compojure;1.1.3: not found [warn] :: hiccup#hiccup;0.3.6: not found [warn] :: ring#ring-devel;0.3.11: not found [warn] :: ring#ring-jetty-adapter;0.3.11: not found [warn] :: com.twitter#carbonite;1.4.0: not found [warn] :: cascading#cascading-hadoop;2.5.6: not found [warn] :::::::::::::::::::::::::::::::::::::::::::::: [trace] Stack trace suppressed: run last *:update for the full output. [error] (*:update) sbt.ResolveException: unresolved dependency: cascading#cascading-local;2.5.6: not found [error] unresolved dependency: clj-time#clj-time;0.4.1: not found [error] unresolved dependency: compojure#compojure;1.1.3: not found [error] unresolved dependency: hiccup#hiccup;0.3.6: not found [error] unresolved dependency: ring#ring-devel;0.3.11: not found [error] unresolved dependency: ring#ring-jetty-adapter;0.3.11: not found [error] unresolved dependency: com.twitter#carbonite;1.4.0: not found [error] unresolved dependency: cascading#cascading-hadoop;2.5.6: not found [error] Total time: 41 s, completed Nov 19, 2014 8:44:04 PM 

My build.sbt as follows

 name := "twitter-sparkstreaming-elasticsearch" version := "0.0.1" scalaVersion := "2.10.4" // additional libraries libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % "1.1.0", "org.apache.spark" %% "spark-streaming" % "1.1.0", "org.apache.spark" %% "spark-streaming-twitter" % "1.1.0", "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.Beta2", "org.elasticsearch" % "elasticsearch-hadoop-cascading" % "2.1.0.Beta2" ) resolvers += "sonatype-oss" at "http://oss.sonatype.org/content/repositories/snapshots" resolvers += "Typesafe Repo" at "http://repo.typesafe.com/typesafe/releases/" 

Answer 5

Only for Spark support can you use a minimalistic binary. Add to libraryDependencies in build.sbt

following:

 "org.elasticsearch" % "elasticsearch-spark_2.10" % "2.1.0.Beta3" 

Note: "2.10" refers to a compatible version of Scala!

And delete

 "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.Beta3" 

This will avoid the unresolved divisions listed in the question.