All geek questions in one place

Hive 0.14 UDF Maven Project Missing Dependencies

I tried to set up a Maven project that would contain user-defined functions (UDFs) that I would like to use in my usage requests. I started with a Maven project that didn't contain source files, and the following POM:

<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>exp</groupId> <artifactId>HiveUdfTestProject</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <maven.compiler.source>1.7</maven.compiler.source> <maven.compiler.target>1.7</maven.compiler.target> </properties> <dependencies> <dependency> <groupId>org.apache.hive</groupId> <artifactId>hive-exec</artifactId> <version>0.14.0</version> </dependency> </dependencies> </project>

When I tried to build the project, I get the following error:

The goal in the HiveUdfTestProject project could not be completed: the dependencies for the exp project could not be determined: HiveUdfTestProject: jar: 1.0-SNAPSHOT: The following artifacts may not be allowed: org.apache.calcite: calcite core: jar: 0.9.2-Incubation-SNAPCHOT, org.apache.calcite: calcite-avatica: jar: 0.9.2-Incubation-SNAPCHOT: Could not find artifact org.apache.calcite: calcite-core: jar: 0.9.2-incubating-SNAPSHOT → [Help 1]

+5
source share
3 answers

I found a cube with a calcite core in the central maven repository (but not in the incubation-snapshot version), the required hive is exec 0.14.0.

Adding a calcite core from the maven center saved the original error and introduced a new missing dependency "pentaho-aggdesigner-algorithm" that I found on ConJars .

Adding concages repo and pentaho dependency on a new missing dependency: org.apache.calcite: calcite-avatica: jar: 0.9.2-incubating-SNAPSHOT ", whose incubation (but not instantaneous) dependency was available in the central maven repository .

Adding calcite-avatica dependencies to the POM led to the successful completion of an empty project.

Here is the final POM needed to create a project designed to build Hive UDF:

 <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>exp</groupId> <artifactId>HiveUdfTestProject</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <maven.compiler.source>1.7</maven.compiler.source> <maven.compiler.target>1.7</maven.compiler.target> </properties> <repositories> <repository> <id>conjars.org</id> <url>http://conjars.org/repo</url> </repository> </repositories> <dependencies> <!-- From Maven Central --> <dependency> <groupId>org.apache.hive</groupId> <artifactId>hive-exec</artifactId> <version>0.14.0</version> </dependency> <dependency> <groupId>org.apache.calcite</groupId> <artifactId>calcite-core</artifactId> <version>0.9.2-incubating</version> </dependency> <dependency> <groupId>org.apache.calcite</groupId> <artifactId>calcite-avatica</artifactId> <version>0.9.2-incubating</version> </dependency> <!-- From conjars --> <dependency> <groupId>org.pentaho</groupId> <artifactId>pentaho-aggdesigner-algorithm</artifactId> <version>5.1.3-jhyde</version> </dependency> </dependencies> </project>

As soon as the empty project was built, I tried to integrate the POM settings into the larger existing Maven project and saw errors related to the calcite kernel that were specifically looking for the snapshot version. To get past this, I changed the hive-exec dependency to look like this:

 <dependency> <groupId>org.apache.hive</groupId> <artifactId>hive-exec</artifactId> <version>0.14.0</version> <exclusions> <exclusion> <groupId>org.apache.calcite</groupId> <artifactId>*</artifactId> </exclusion> </exclusions> </dependency>
+3
source

I explicitly included calcite-core and calcite-avatica projects as dependencies, and my project (which also includes Hive 14 dependencies) no longer failed because "artifacts could not be fixed"

  <dependency> <groupId>org.apache.calcite</groupId> <artifactId>calcite-core</artifactId> <version>1.0.0-incubating</version> </dependency> <dependency> <groupId>org.apache.calcite</groupId> <artifactId>calcite-avatica</artifactId> <version>1.0.0-incubating</version> </dependency>

From what I can tell, this is an open problem with Hive 14. For more information see https://issues.apache.org/jira/browse/HIVE-8906 .

+2
source

I solved the same problem by adding below dependencies to /ql/pom.xml org.pentaho Pentaho-aggdesigner algorithm 5.1.3-jhyde

 <dependency> <groupId>eigenbase</groupId> <artifactId>eigenbase-properties</artifactId> <version>1.1.4</version> </dependency> <dependency> <groupId>net.hydromatic</groupId> <artifactId>linq4j</artifactId> <version>0.4</version> </dependency>

and below the repository in /pom.xml under the positions

 <repository> <id>conjars</id> <name>Concurrent Conjars repository</name> <url>http://conjars.org/repo</url> <layout>default</layout> </repository>
0
source

Source: https://habr.com/ru/post/1210118/

More articles:


All Articles