I am trying to execute a shell script through oozie, but I am having some problems.
I have a properties file similar to this (import.properties):
startIndex=2000 chunkSize=2000
The idea is that with each execution, the value of startIndex will be updated with the block size. Therefore, if I performed it, it should have
startIndex=4000 chunkSize=2000
I tested the script separately and it works great. Here are my other related files.
job.properties
nameNode=hdfs://192.168.56.101:8020 jobTracker=192.168.56.101:50300 wfeRoot=wfe queueName=default EXEC=script.sh propertyLoc=import.properties oozie.use.system.libpath=true oozie.wf.application.path=${nameNode}/user/${user.name}/${wfeRoot}/coordinator
workflow.xml
<workflow-app xmlns='uri:oozie:workflow:0.2' name='shell-wf'> <start to='shell1' /> <action name='shell1'> <shell xmlns="uri:oozie:shell-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <exec>${EXEC}</exec> <file>${EXEC}#${EXEC}</file> <file>${propertyLoc}#${propertyLoc}</file> </shell> <ok to="end" /> <error to="fail" /> </action> <kill name="fail"> <message>Script failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name='end' />
script.sh
#!/bin/sh file=import.properties . $file SCRIPT=$(readlink -f $file) SCRIPTPATH=$(dirname $SCRIPT) echo $SCRIPTPATH newStartIndex=`expr $chunkSize + $startIndex` newStartIndexStr=startIndex=$newStartIndex oldStartIndexStr=startIndex=$startIndex chunkSizeStr=chunkSize=$chunkSize sed -i "s|$oldStartIndexStr|$newStartIndexStr|g" $file
And I put all these files in the HDFS working directory:
[ ambari_qa@sandbox coordinator]$ hadoop fs -lsr /user/ambari_qa/wfe/coordinator -rw-rw-rw- 1 ambari_qa hdfs 32 2013-05-09 00:12 /user/ambari_qa/wfe/coordinator/import.properties -rw-rw-rw- 1 ambari_qa hdfs 533 2013-05-09 01:19 /user/ambari_qa/wfe/coordinator/script.sh -rw------- 1 ambari_qa hdfs 852 2013-05-09 00:50 /user/ambari_qa/wfe/coordinator/workflow.xml
I expected the import.properties file to be changed after each execution. But I see that this does not change, although the work of Oozi is successful. For the purpose of debugging, I printed the location of the file at runtime and found that it was copied to another location (from the log):
>>> Invoking Shell command line now >> Stdoutput /hadoop/mapred/taskTracker/ambari_qa/distcache/-5756672768810005023_889271025_125659265/192.168.56.101/user/ambari_qa/wfe/coordinator Stdoutput startIndex=4000 Stdoutput startIndex=2000 Exit code of the Shell command 0 <<< Invocation of Shell command completed <<<
What do I need to do to make it work with the HDFS working directory? Thanks in advance.
Update:
After changing the script based on Chris's suggestion, it becomes (last 3 lines):
hadoop fs -rm hdfs://ip-10-0-0-92:8020/user/ambari_qa/wfe/shell-oozie/$file sed -i "s|$oldStartIndexStr|$newStartIndexStr|g" $file hadoop fs -put $file /user/ambari_qa/wfe/shell-oozie
But then I began to run into a resolution problem. I gave permission to write to this file and folder.
[ ambari_qa@ip-10-0-0-91 shell-oozie]$ hadoop fs -ls /user/ambari_qa/wfe/shell-oozie
Found 3 items:
-rw-rw-rw- 3 ambari_qa hdfs 32 2013-05-10 16:55 /user/ambari_qa/wfe/shell-oozie/import.properties -rw-rw-rw- 3 ambari_qa hdfs 540 2013-05-10 16:48 /user/ambari_qa/wfe/shell-oozie/script.sh -rw-rw-rw- 3 ambari_qa hdfs 826 2013-05-10 15:29 /user/ambari_qa/wfe/shell-oozie/workflow.xml
Here is the error log:
rm: org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=EXECUTE, inode="ambari_qa":ambari_qa:hdfs:rwxrwx--- put: org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=EXECUTE, inode="ambari_qa":ambari_qa:hdfs:rwxrwx--- Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]