Spark job SBT Merge Merge Conflict Merge When Power On Spark Streaming Kinesis ASL library

Question

I started doing work with spark flow and got a producer for the kinesite endpoint. Having received this work, I began to make a consumer, but I had problems with construction.

I use the build plugin to create a single jar containing all the dependencies. The project dependencies are as follows.

libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.4.1" % "provided",
"org.apache.spark" %% "spark-sql" % "1.4.1" % "provided",
"org.apache.spark" %% "spark-streaming" % "1.4.1" % "provided",
"org.apache.spark" %% "spark-streaming-kinesis-asl" % "1.4.1",
"org.scalatest" %% "scalatest" % "2.2.1" % "test",
"c3p0" % "c3p0" % "0.9.1.+",
"com.amazonaws" % "aws-java-sdk" % "1.10.4.1",
"mysql" % "mysql-connector-java" % "5.1.33",
"com.amazonaws" % "amazon-kinesis-client" % "1.5.0"

)

When I run the build, the files may compile, but the merge phase fails with an error

[error] (streamingClicks/*:assembly) deduplicate: different file contents found in the following:
[error] /Users/adam/.ivy2/cache/org.apache.spark/spark-network-common_2.10/jars/spark-network-common_2.10-1.4.1.jar:META-INF/maven/com.google.guava/guava/pom.properties
[error] /Users/adam/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:META-INF/maven/com.google.guava/guava/pom.properties

This is due to the addition of a spark-kinesite flow dependence. How do I get around this? I can mark the dependency as provided, but then add the jar to the classpath, but this is really not what I want to do.

Answer 1

, . - :

lazy val strategy = assemblyMergeStrategy in assembly <<= (assemblyMergeStrategy in assembly) { (old) => {
  case "application.conf" => MergeStrategy.concat
  case meta(_) => MergeStrategy.discard
  case x => MergeStrategy.first
}
}
lazy val core = (project in file("core")).
  settings(
    name := "core",
    libraryDependencies ++= Seq(
     ...
    ),
    strategy
  )

P.S. "fatjar" . fatjar hdfs sbt package. , fatjar --jar.