All geek questions in one place

Storm command does not work with NoClassDefFoundError after adding jsoup according to specified dependency

I use JSoup in my project and I declared a dependency in my POM file. It compiles just fine and works fine, but only when I used jar with all dependencies and changed the scope to compiled .

If I change this area to provided , then I can still compile it just fine, but not run it. This gives me a ClassNotFoundException . I have included the required JAR file in classpath variables as well as path , but I am still facing this problem.

I can work with the compilation option, but it really annoys me in the depths of my mind why I cannot get it to work with the provided option, and I would really appreciate it if someone can help me understand why.

The following is the error that I see:

 java.lang.NoClassDefFoundError: Lorg/jsoup/nodes/Document; at java.lang.Class.getDeclaredFields0(Native Method) at java.lang.Class.privateGetDeclaredFields(Class.java:2300) at java.lang.Class.getDeclaredField(Class.java:1882) at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1605) at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:50) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:423) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:411) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1114) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330) at backtype.storm.utils.Utils.serialize(Utils.java:52) at backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:94) at com.yahoo.amit.wordstorm.WordStormTopology.main(WordStormTopology.java:25) Caused by: java.lang.ClassNotFoundException: org.jsoup.nodes.Document at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) ... 14 more

Below is my POM file:

 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.yahoo.amit.wordstorm</groupId> <artifactId>wordstorm</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <name>wordstorm</name> <url>http://maven.apache.org</url> <repositories> <repository> <id>clojars.org</id> <url>http://clojars.org/repo</url> </repository> </repositories> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> </properties> <dependencies> <dependency> <groupId>storm</groupId> <artifactId>storm</artifactId> <version>0.8.2</version> <scope>provided</scope> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>3.8.1</version> <scope>test</scope> </dependency> <dependency> <groupId>org.jsoup</groupId> <artifactId>jsoup</artifactId> <version>1.7.2</version> <scope>provided</scope> </dependency> </dependencies> <build> <plugins> <!-- bind the maven-assembly-plugin to the package phase this will create a jar file without the storm dependencies suitable for deployment to a cluster. --> <plugin> <artifactId>maven-assembly-plugin</artifactId> <configuration> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> <archive> <manifest> <mainClass></mainClass> </manifest> </archive> </configuration> <executions> <execution> <id>make-assembly</id> <phase>package</phase> <goals> <goal>single</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </project>

Following are my system variables:

 echo $PATH /Users/programmerman/Summer Project/apache-maven-3.0.5/bin/:/Users/programmerman/Summer Project/storm-0.8.2/bin/:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Users/programmerman/Summer Project/CLASSPATH/jsoup-1.7.2.jar:/Users/programmerman/Summer Project/CLASSPATH/* echo $CLASSPATH /Users/programmerman/Summer Project/storm-0.8.2/storm-0.8.2.jar:/Users/programmerman/Summer Project/storm-0.8.2/lib/*:/Users/programmerman/Summer Project/storm-0.8.2/conf/storm.yaml:/Users/programmerman/SummerProject/storm-starter-masterPOM/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar:/Users/programmerman/Summer Project/CLASSPATH/jsoup-1.7.2.jar:/Users/programmerman/Summer Project/CLASSPATH/*
+4
source share
3 answers

This is a question about Maven as a Storm and its deployment model. You should check what the storm command does . First of all, this is actually a Python script, which eventually calls java .

If you look at the get_classpath(extrajars) function, you will notice that it does not use the $CLASSPATH evironment variable at all. Rather, it loads the Storm core and any banks that you have in the lib/ directory relative to your working directory, as well as the configuration files in the ~/.storm

(You will find that ignoring $CLASSPATH very common in many Java applications. Usually the first thing that a “script run” does is to overwrite CLASSPATH or not to use it at all. This is to prevent unknown / unsupported / earlier versions of your jars due to problems in your application).

As for your application, when jsoup is declared "provided": when you declare a bank as a provided dependency, it will not be packaged in the "jar with dependencies" assembly. See this question for a good explanation: The difference between maven scope compilation and provided for JAR packaging

Explanation tl; dr is that the compilation area comes with your uber-jar, provided that the area does not exist, since it must be "provided" by the container into which you are deploying. Typically, a “container” is a Java web server such as Tomcat (therefore, you will never have to send JSPs or servlets with your Java web applications). In this case, the “container” that you expect to provide is Storm. However, jsoup is not provided by Storm, hence your mistake.

Warehouses of compilation classes should still be sent along with your application, because your application will create / use interfaces, enumerations, etc.

My recommendation is to simply declare the jsoup scope to “compile” and move on. An alternative would be to write your own individual script deployment and / or assembly that puts jsoup under lib/ - essentially the same thing at the end.

+2
source

The Storm script does not use the CLASSPATH variable, but instead puts all the banks in the STORM_DIR / lib directory in its class path. So you have 2 options:

  • Change the JSoup dependency region to “compile” and pack it in the dependency jar.
  • Leave the JSoup dependency in the “provided” area and copy the JSoup jar into the STORM_DIR / lib directory so that the storm script automatically places this jar in its classpath.

I highly recommend option 1 and follow Maven's standard approaches.

Just FYI, this is how a Storm script creates a classpath line:

 def get_classpath(extrajars): ret = get_jars_full(STORM_DIR) ret.extend(get_jars_full(STORM_DIR + "/lib")) ret.extend(extrajars) return normclasspath(":".join(ret))
+2
source

The Maven scope shown here means that dependencies are included in the compilation, but not at runtime. It is expected that the container / script run will include it explicitly, so I can see that you are on the right track.

Other issues you can check to fix the problem:

  • Check the CLASSPATH environment variable on the shell instance that runs the java program. Although you already have the correct CLASSPATH value in your custom shell, you often have to create a new shell instance (for example: running a script), and the CLASSPATH variable is not propagated. On UNIX, this is usually done using the export command.
  • Check that the path to the class path is valid, has the correct file system permission, the banks are not damaged.
  • Check out the java command you used to run the program. If you specify -cp , it may (or not) override the CLASSPATH environment variable
0
source

Source: https://habr.com/ru/post/1485719/

More articles:


All Articles