Java.lang.NoClassDefFoundError: kafka / common / TopicAndPartition

Question

When executing the following command in my code:

kafka_streams = [KafkaUtils.createStream(ssc, zk_settings['QUORUM'], zk_settings['CONSUMERS'][k], {zk_settings['TOPICS'][0]: zk_settings['NUM_THREADS']}) .window(zk_settings['WINDOW_DURATION'], zk_settings['SLIDE_DURATION']) for k in range(len(zk_settings['CONSUMERS']))] 

but I get the following error:

 Exception in thread "Thread-3" java.lang.NoClassDefFoundError: kafka/common/TopicAndPartition at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2625) at java.lang.Class.privateGetPublicMethods(Class.java:2743) at java.lang.Class.getMethods(Class.java:1480) at py4j.reflection.ReflectionEngine.getMethodsByNameAndLength(ReflectionEngine.java:365) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:317) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:342) at py4j.Gateway.invoke(Gateway.java:252) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:207) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ClassNotFoundException: kafka.common.TopicAndPartition at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 12 more 

Did I miss something?

I was getting a spark error, so I restored the spark error, and this led to this error.

Answer 1

When submitting the code, you must add --packages .

  ./bin/spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.2.0 <DIR>/main.py localhost:9092 test 

https://spark.apache.org/docs/latest/streaming-kafka-0-8-integration.html

Answer 2

classNotFoundException means that your spark-submit program is running; it could not find the required class kafka.common.TopicAndPartition in the current directory of the program.

take a look at using the spark-submit command:

 # spark-submit --help Usage: spark-submit [options] <app jar | python file> [app arguments] Usage: spark-submit --kill [submission ID] --master [spark://...] Usage: spark-submit --status [submission ID] --master [spark://...] Usage: spark-submit run-example [options] example-class [example args] Options: --master MASTER_URL spark://host:port, mesos://host:port, yarn, or local. --deploy-mode DEPLOY_MODE Whether to launch the driver program locally ("client") or on one of the worker machines inside the cluster ("cluster") (Default: client). --class CLASS_NAME Your application main class (for Java / Scala apps). --name NAME A name of your application. --jars JARS Comma-separated list of local jars to include on the driver and executor classpaths. --packages Comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. Will search the local maven repo, then maven central and any additional remote repositories given by --repositories. The format for the coordinates should be groupId:artifactId:version. 

Add the -jars parameter with the local jar kafka course as follows:

# spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.2.0 --jars /path/to/org.apache.kafka_kafka_2.11-0.8.2.1.jar,/path/to/com.yammer.metrics_metrics-core-2.2.0.jar your_python_script.py .