All geek questions in one place

Hadoop library conflicts in mapreduce time

I have a jar that uses the Hadoop API to run various remote mapreduce jobs (i.e. they don’t use the command line to run the job). A service block that performs various tasks is built using maven "jar-with-dependencies".

My tasks run fine, except for one using commons-codec 1.7, I get:

FATAL org.apache.hadoop.mapred.Child: Error starting child: java.lang.NoSuchMethodError: org.apache.commons.codec.binary.Base64.encodeAsString ([B) Ljava / lang / String;

I think this is due to the fact that my bank has commons-codec 1.7, while my Hadoop install lib has commons-codec 1.4 ...

Is there any way they could instruct Hadoop to use distributed commons-codec 1.7 (I assume this spreads as a dependency on work), and not commons-codec 1.4 in hasoop 1.0.3 core lib?

Many thanks!

Note. Removing commons-codec-1.4.jar from my Hadoop library folder does solve the problem, but doesn't seem too sane. Hope there is a better alternative.

+1
source share
1 answer

Two approaches:

  • You should be able to exclude the public codec from hadoop dependencies and add another explicit dependency for commons-codec
  • Try setting the scope so that none of the storage cans are turned on. This suggests that these banks will be in the path of the execution class.
0
source

Source: https://habr.com/ru/post/1207530/

More articles:


All Articles