Oozie> Spark action> why the jar element doesn't accept multiple jars

Question

I am new to Spark judicial action.

As mentioned on the Oozie Spark action page: https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html

The jar element indicates a comma-separated list of jars or python files.

so I added sth like: test1.jar, test2.jar

But the problem is that Ozie does not recognize her at all. Anyway, if I add only one jar to the jar element, it really works.

Why is Oozie not working as shown on the document website?

Thanks in advance!

More comments: after a long study, I really feel that Oozie is not easy to use, but just takes a lot of time to study it. Anyone have suggestions on some other tools that do the same job on a Hadoop system?

Answer 1

I have the same problem and can be solved this way:

Define in the properties:

libDir=${nameNode}${hdfsHomeDir}/lib
sparkLib=${libDir}/sparklib.jar,${libDir}/spark-csv_2.10-1.3.0.jar,${libDir}/commons-csv-1.1.jar

Then

<action name="activeuser">
       <spark xmlns="uri:oozie:spark-action:0.1">
           <jar>${nameNode}${hdfsHomeDir}/lib/${sparkActiveUserJarExe}</jar>
           <spark-opts>--jars ${sparkLib}</spark-opts>
        </spark>
        <ok to="end"/>
        <error to="wl_fail"/>
    </action>