Getting exception: java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror (Ljava / lang / ClassLoader;) when using data frames

Question

I get the message "java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror (Ljava / lang / ClassLoader;)" when using data files in a scala application and start it using a spark. However, if I work using only RDD and not data frames, such an error does not occur with the same parameters and settings. Also, when viewing other messages with the same error, it is mentioned that the scala version should be 2.10, since the spark is not compatible with 2.11 scala, and I use version 2.10 scala with spark 2.0.0.

Below is the snip from pom: <properties> <spark-assembly>/usr/lib/spark/lib/spark-assembly.jar</spark-assembly> <encoding>UTF-8</encoding> <hadoop.version>2.7.1</hadoop.version> <hbase.version>1.1.1</hbase.version> <scala.version>2.10.5</scala.version> <scala.tools.version>2.10</scala.tools.version> <spark.version>2.0.0</spark.version> <phoenix.version>4.7.0-HBase-1.1</phoenix.version> </properties> <dependencies> <dependency> <groupId>org.scala-lang</groupId> <artifactId>scala-library</artifactId> <version>${scala.version}</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>${hadoop.version}</version> </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-client</artifactId> <version>${hbase.version}</version> </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-server</artifactId> <version>${hbase.version}</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_${scala.tools.version}</artifactId> <version>${spark.version}</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_${scala.tools.version}</artifactId> <version>${spark.version}</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-hive_${scala.tools.version}</artifactId> <version>${spark.version}</version> </dependency> </dependencies> 

Error:

 16/10/19 02:57:26 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror; java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror; at com.abc.xyz.Compare$.main(Compare.scala:64) at com.abc.xyz.Compare.main(Compare.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:627) 16/10/19 02:57:26 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror;) 16/10/19 02:57:26 INFO spark.SparkContext: Invoking stop() from shutdown hook 

Answer 1

Change the scala version

<scala.version>2.11.8</scala.version >

 <scala.tools.version>2.11</scala.tools.version> 

and add

 <dependency> <groupId>org.scala-lang</groupId> <artifactId>scala-reflect</artifactId> <version>${scala.version}</version> </dependency> 

Answer 2

I also ran into this error and this is purely a version issue.

Your scala version is not compatible or you are probably using the correct version, but intellij libraries have an old version.

Quick fix:

I use spark 2.2.0 and scala 2.10.4, which I then changed to scala version 2.11.8. After that do the following:

1) right-click on the intellij module 2) open the settings module. 3) go to libraries and clean them all 4) restore

By doing the above for me the problem is solved.