All geek questions in one place

Oozie continues to add old version of httpcore jar to classpath

I get an exception because Oozie is adding the wrong version of the httpcore jar to the classpath. I tried various options like

oozie.launcher.mapreduce.task.classpath.user.precedence oozie.launcher.mapreduce.user.classpath.first

oozie.launcher.mapreduce.task.classpath.user.precedence is not responding at all, and when I use oozie.launcher.mapreduce.user.classpath.first, the application cannot load even one class.

In the class path, I see two versions of http-core.

 httpcore-4.4.1.jar httpcore-4.2.4.jar

When the application launches offline, I do not get this exception.

An exception:

  Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], main() threw exception, java.lang.NoSuchFieldError: INSTANCE org.apache.oozie.action.hadoop.JavaMainException: java.lang.NoSuchFieldError: INSTANCE at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47) at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.NoSuchFieldError: INSTANCE at org.apache.http.conn.ssl.SSLConnectionSocketFactory.<clinit>(SSLConnectionSocketFactory.java:144) at microsoft.exchange.webservices.data.core.ExchangeServiceBase.createConnectionSocketFactoryRegistry(ExchangeServiceBase.java:244) at microsoft.exchange.webservices.data.core.ExchangeServiceBase.initializeHttpClient(ExchangeServiceBase.java:198) at microsoft.exchange.webservices.data.core.ExchangeServiceBase.<init>(ExchangeServiceBase.java:174) at microsoft.exchange.webservices.data.core.ExchangeServiceBase.<init>(ExchangeServiceBase.java:179) at microsoft.exchange.webservices.data.core.ExchangeService.<init>(ExchangeService.java:3729) at com.sonasoft.sonacloud.email.dispatcher.conn.EwsConnection.getConnection(EwsConnection.java:16) at com.sonasoft.sonacloud.email.dispatcher.conn.EwsConnection.getConnection(EwsConnection.java:10) at com.sonasoft.sonacloud.email.dispatcher.utils.EwsOperations.<init>(EwsOperations.java:47) at com.sonasoft.sonacloud.email.dispatcher.utils.EwsOperations.getInstance(EwsOperations.java:53) at com.sonasoft.sonacloud.email.dispatcher.main.MainClass.main(MainClass.java:41) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56) ... 15 more

Oozie client build version: 4.2.0.2.3.2.0-2950

Any help is appreciated.

+5
source share
2 answers

We had this nasty problem with HortonWorks distro 2.3.2 (ashamed of them):

  • Oozi's “launch” task always gets httpcore and httpclient in CLASSPATH as part of the Hadoop client
  • Oozi's “launch” institution always receives httpcore and httpclient as specified in Oozi's ShareLib
  • Hive / Hive2 Sharelibs contain httpcore and httpclient in a later version
  • in terms of Hadoop, user.classpath.first applies to both ShareLibs, so there’s a 50/50 chance of getting the right order for each JAR (so the probability is 25/75 overall)

Bottom line: we had to

  • remove httpcore and httpclient from file "Oozie" ShareLib HDFS dir (duh!)
  • flag oozie.launcher.mapreduce.job.user.classpath.first for all Hive JARS-based actions (i.e. Hive action, Hive2 action, shell action calling the JDBC driver in some way, etc.) .

Post-scriptum - The Oozie server stores a list of JARs in each ShareLib, so deleting the JAR while the server is running will result in errors in new jobs. If you do not want to stop the Oozie server, then the “right way” to update live ShareLib is to (a) create a new version in a new directory with a time stamp [check the documentation ...] and (b) tell the server to re-synchronize the new libraries with using oozie admin -sharelibupdate

+3
source

You want to create your local version of the httpcore JAR, but you do not want it to be present on your class path, because Hadoop will provide its own version. Then you should use the provided scope for the httpcore JAR:

 <project> ... <dependencies> <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpcore</artifactId> <scope>provided</scope> <!-- this line is important --> <version>4.4.1</version> </dependency> </dependencies> </project>

From the Maven documentation for provided :

This is very similar to compilation, but indicates that you expect the JDK or container to provide a dependency at runtime.

0
source

Source: https://habr.com/ru/post/1240895/

More articles:


All Articles