Oozie Java Action: Passing an Hbase Class Path

Question

I am running a hbase test java program through the oozie java action. The following error has occurred:

Failing Oozie Launcher, Main class [HbaseTest], main() threw exception, org/apache/hadoop/hbase/HBaseConfiguration java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration at HbaseTest.main(HbaseTest.java:28) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:495) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) ... 14 more 

The program works correctly from the command line:

 java -cp `hbase classpath` HbaseTest 

Is there a way to pass the result of the 'hbase classpath' to the oozie java action. I do not want to copy jbase jars to the workflow lib directory, as this will be overhead maintenance.

The following is the java action from workflow.xml :

  <java> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <main-class>HbaseTest</main-class> <java-opts></java-opts> <arg>HELLO</arg> </java> 

Answer 1

Since Oozie 2.3 you can use Share Libraries:

Oozie maintains a partition library for work and a system for work tasks.

Shared libraries can simplify the deployment and management of shared components in workflow applications.

For example, if a workflow job uses a shared library with Streaming, Pig, and Har JAR files, it does not need to link these JAR files in the folder / path of the workflow application.

If the workflow job uses a partition library, Oozie will include all the JAR / SO files in the library in the classpath / libpath for all of its actions.

The workflow task can specify the path to the shared library using the job property oozie.libpath.

A workflow task can use the system’s shared resources library by setting the job property oozie.use.system.libpath to true.

http://oozie.apache.org/docs/3.2.0-incubating/WorkflowFunctionalSpec.html#a17_HDFS_Share_Libraries_for_Workflow_Applications_since_Oozie_2.3

How to install: http://oozie.apache.org/docs/4.0.0/DG_QuickStart.html#OozieShareLib