FlinkMLTools NoClassDef when starting a jar built with maven

Question

I am working on a recommendation system using Apache Flink. The implementation is done when I test it in IntelliJ, but now I would like to switch to a cluster. I also created a jar file and tested it locally to see if everything worked, but I ran into a problem.

java.lang.NoClassDefFoundError: org / apache / flink / ml / common / FlinkMLTools $

As we can see, the class FlinkMLToolsused in my code was not found while jar was running. I built this jar with Maven 3.3.3 using mvn clean install, and I am using version 0.9.0 for Flink.

First trail

The fact is that my global project contains other projects (and this recommender is one of the subprojects). Thus, I have to start mvn clean installthe folder of the desired project, otherwise Maven always creates a jar of another project (and I do not understand why). So I'm wondering if there could be a way to explicitly tell maven to create one specific project of a global project. In fact, perhaps the path to is FlinkMLToolscontained in the link present in the pom.xmlglobal project file .

Any other ideas?

Answer 1

, Flink (flink-ml, gelly ..). , jar , . .

,

, , - Flink pom.

mvn archetype:generate -DarchetypeGroupId=org.apache.flink \
-DarchetypeArtifactId=flink-quickstart-scala -DarchetypeVersion=0.9.0 

Flink, API Scala. pom .

<dependencies>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-scala</artifactId>
        <version>0.9.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-streaming-scala</artifactId>
        <version>0.9.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-clients</artifactId>
        <version>0.9.0</version>
    </dependency>
</dependencies>

flink-streaming-scala, , Flink.

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-ml</artifactId>
    <version>0.9.0</version>
</dependency>

, mvn package, flink-ml .

Flink , <FLINK_ROOT_DIR>/lib . , Flink, flink-ml jar /lib. , , , , , all.

maven

X , :

 mvn clean package -pl X -am

-pl , , -am maven . .

Answer 2

Flink JAR . IntelliJ , .

:

• FlinkML Jar lib Flink TaskManager
• Jar , FLinkML.

. .