NoSuchMethodError when running on Hadoop, but not when running locally

Question

When starting the program on Hadoop 2.0.0-cdh4.3.1, MapReduce gives me the error below:

java.lang.NoSuchMethodError:com.google.common.util.concurrent.Futures.withFallback 

But when I test by doing a JAR:

 java -cp myclass 

It works flawlessly. I don’t understand here, as if the so-called Futures.withFallback is present in the JAR, and therefore it was executed in local mode. Its using Guava to connect Cassandra, a full stack trace below:

 attempt_201507081740_21115_m_000050_0: [FATAL] Child - Error running child : java.lang.NoSuchMethodError: com.google.common.util.concurrent.Futures.withFallback(Lcom/google/common/util/concurrent/ListenableFuture;Lcom/google/common/util/concurrent/FutureFallback;Ljava/util/concurrent/Executor;)Lcom/google/common/util/concurrent/ListenableFuture; attempt_201507081740_21115_m_000050_0: at com.datastax.driver.core.Connection.initAsync(Connection.java:176) attempt_201507081740_21115_m_000050_0: at com.datastax.driver.core.Connection$Factory.open(Connection.java:721) attempt_201507081740_21115_m_000050_0: at com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:244) attempt_201507081740_21115_m_000050_0: at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:190) attempt_201507081740_21115_m_000050_0: at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:78) attempt_201507081740_21115_m_000050_0: at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1272) attempt_201507081740_21115_m_000050_0: at com.datastax.driver.core.Cluster.init(Cluster.java:158) attempt_201507081740_21115_m_000050_0: at com.datastax.driver.core.Cluster.connect(Cluster.java:248) attempt_201507081740_21115_m_000050_0: at com.datastax.driver.core.Cluster.connect(Cluster.java:281) attempt_201507081740_21115_m_000050_0: at com.cassandra.CassandraHandler.getConnection(CassandraHandler.java:40) attempt_201507081740_21115_m_000050_0: at com.json.flatten.DynamicJsonFlattener.<init>(DynamicJsonFlattener.java:35) attempt_201507081740_21115_m_000050_0: at com.mapreduce.Map.map(Map.java:18) attempt_201507081740_21115_m_000050_0: at com.mapreduce.Map.map(Map.java:13) attempt_201507081740_21115_m_000050_0: at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140) attempt_201507081740_21115_m_000050_0: at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) attempt_201507081740_21115_m_000050_0: at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) attempt_201507081740_21115_m_000050_0: at org.apache.hadoop.mapred.Child$4.run(Child.java:268) attempt_201507081740_21115_m_000050_0: at java.security.AccessController.doPrivileged(Native Method) attempt_201507081740_21115_m_000050_0: at javax.security.auth.Subject.doAs(Subject.java:396) attempt_201507081740_21115_m_000050_0: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) attempt_201507081740_21115_m_000050_0: at org.apache.hadoop.mapred.Child.main(Child.java:262) 

Something is wrong with the version of Hadoop or any other version. Any ideas please!

EDIT: I checked that there is no "withFallback" method in the Guava v18 JAR. Now I don’t know, please help me with any ideas!

Answer 1

Edited: did not see your stack. your stacktrace shows that there is a version match. You can find the corresponding compatible version of the documentation there.

make sure the required jar files are available in your classpath.

when you use the -cp option, your jvm looks for the desired jar file in all directories separated by; defined in the classpath variable.

make sure you also add ;;; in your classpath, which causes jvm to load / search for class / jar files available in your current working directory.

Answer 2

I believe I have found a problem (or at least a resolution for me).

If you use google-api-client library:

com.google.api-client » google-api-client

it has a dependency on guava-jdk5 13.0 . This version of guava-jdk5 does not have Futures.withFallback (which was introduced in 14.0 ) and will probably contradict your dependency on guava.

The solution, if this is your case, is to add a dependency to com.google.guava:guava-jdk5:17.0 . This is a JDK5 backport guava , but it works to solve this problem.

Why Google has this dependency on a very old version of guava (and nonetheless backport) is outside of me.