All geek questions in one place

Spark 1.4.0 java.lang.NoSuchMethodError: com.google.common.base.Stopwatch.elapsedMillis () J

I use spark 1.4.0 / hadoop 2.6.0 (only for hdfs) and when running the Scala example SparkPageRank (examples / src / main / scala / org / apache / spark / examples / SparkPageRank.scala), I encounter the following error:

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Stopwatch.elapsedMillis()J at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:245) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:207) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) at org.apache.spark.rdd.RDD$$anonfun$distinct$2.apply(RDD.scala:329) at org.apache.spark.rdd.RDD$$anonfun$distinct$2.apply(RDD.scala:329) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108) at org.apache.spark.rdd.RDD.withScope(RDD.scala:286) at org.apache.spark.rdd.RDD.distinct(RDD.scala:328) at org.apache.spark.examples.SparkPageRank$.main(SparkPageRank.scala:60) at org.apache.spark.examples.SparkPageRank.main(SparkPageRank.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:621) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

I am not very familiar with java, but it seems like this is a guava problem

The following information could be helpfup:

 $ find ./spark -name *.jars | grep guava ./lib_managed/bundles/guava-16.0.1.jar ./lib_managed/bundles/guava-14.0.1.jar

part of the examples / pom.xml file:

 ... <dependency> <groupId>org.apache.cassandra</groupId> <artifactId>cassandra-all</artifactId> <version>1.2.6</version> <exclusions> <exclusion> <groupId>com.google.guava</groupId> <artifactId>guava</artifactId> </exclusion> ...

And indeed, it seems that the class does not contain the problematic method:

 $ javap -p /mnt/spark/examples/target/streams/\$global/assemblyOption/\$global/streams/assembly/7850cb6d36b2a6589a4d27ce027a65a2da72c9df_5fa98cd1a63c99a44dd8d3b77e4762b066a5d0c5/com/google/common/base/Stopwatch.class Compiled from "Stopwatch.java" public final class com.google.common.base.Stopwatch { private final com.google.common.base.Ticker ticker; private boolean isRunning; private long elapsedNanos; private long startTick; public static com.google.common.base.Stopwatch createUnstarted(); public static com.google.common.base.Stopwatch createUnstarted(com.google.common.base.Ticker); public static com.google.common.base.Stopwatch createStarted(); public static com.google.common.base.Stopwatch createStarted(com.google.common.base.Ticker); public com.google.common.base.Stopwatch(); public com.google.common.base.Stopwatch(com.google.common.base.Ticker); public boolean isRunning(); public com.google.common.base.Stopwatch start(); public com.google.common.base.Stopwatch stop(); public com.google.common.base.Stopwatch reset(); private long elapsedNanos(); public long elapsed(java.util.concurrent.TimeUnit); public java.lang.String toString(); private static java.util.concurrent.TimeUnit chooseUnit(long); private static java.lang.String abbreviate(java.util.concurrent.TimeUnit); }

I would like to better understand the problem and, if possible, learn how to fix it :-)

+8
source share
3 answers

The elapsedMilis() method was deleted in Guava 16. (Or it was planned to delete - in any case, there is no method with this name in your list.)

As far as I remember, in guava 16 there should be something like TimeUnit.MILLISECONDS , or you can convert manually by dividing 1000000000.0.

+4
source

Try upgrading Hadoop from 2.6.0 to 2.6.5. In my case, he solved the problem with the stopwatch when using the HBaseAdmin.tableExists function (other dependencies: Hbase 1.2.0, Spark 2.0.1, Scala 2.11.8), although the solution for this problem in Hbase is planned to be released in 1.3.0, which in currently unavailable for production link

0
source

I am using Spark 2.4.4, and still have this problem! In any case, I do not need to see this log output, so I just changed the logging level for the class in question:

  <logger name="org.apache.hadoop.mapred.FileInputFormat" level="INFO" additivity="false"> <appender-ref ref="CONSOLE"/> </logger>
0
source

Source: https://habr.com/ru/post/989693/

More articles:


All Articles