Hadoop Maven Dependency Error

Question

I am trying to build a Hadoop job using Maven. This job works well when I don't use Maven and directly import the Hadoop Jar dependency into eclipse.

I can also create a simple jar (hello world type) using Maven without a Hadoop dependency.

But when I add a Hadoop dependency and run jar, I get this error:

dataException in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration at org.graphhadoop.CreateAdjacency.Adjacency(CreateAdjacency.java:74) at org.graphhadoop.Driver.main(Driver.java:12) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) ... 2 more 

The changes to the pom file are as follows:

  <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>3.8.1</version> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-core</artifactId> <version>0.20.203.0</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-jar-plugin</artifactId> <configuration> <archive> <manifest> <addClasspath>true</addClasspath> <mainClass>org.graphhadoop.Driver</mainClass> </manifest> </archive> <manifestEntries> <Class-Path>../target/classes</Class-Path> </manifestEntries> </configuration> </plugin> </plugins> </build> 

Where am I mistaken?

Edit:

I am currently running Jar locally - not on Hadoop, but on local data.

 java -jar jarname.jar 

Answer 1

Just like a solution for someone facing the same problem, here is what resolved this:

These dependencies need to be added to pom.xml:

  <dependency> <groupId>org.codehaus.jackson</groupId> <artifactId>jackson-mapper-asl</artifactId> <version>1.5.2</version> </dependency> <dependency> <groupId>net.sf.json-lib</groupId> <artifactId>json-lib</artifactId> <version>2.3</version> <classifier>jdk15</classifier> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-core</artifactId> <version>0.20.203.0</version> </dependency> </dependencies> 

and these assembly plugins:

 <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-assembly-plugin</artifactId> <executions> <execution> <goals> <goal>attached</goal> </goals> <phase>package</phase> <configuration> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> <archive> <manifest> <addClasspath>true</addClasspath> <mainClass>org.lll..</mainClass> </manifest> <manifestEntries> <Class-Path>../target/classes</Class-Path> </manifestEntries> </archive> </configuration> </execution> </executions> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-jar-plugin</artifactId> <configuration> <archive> <manifest> <addClasspath>true</addClasspath> <mainClass>org...</mainClass> </manifest> </archive> <manifestEntries> <Class-Path>../target/classes</Class-Path> </manifestEntries> </configuration> </plugin> </plugins> </build>