GNU In parallel in a BASH script with "export -f <func>" with the error "Command not found" when Crond

Question

My script works if I run it interactively in the shell:

$ cat ndmpcopy_cron_parallel_svlinf05.bash #!/usr/software/bin/bash ndmpcopy_cron_parallel() { timestamp=`date +%Y%m%d-%H%M` LOG=/x/eng/itarchives/ndmpcopylogs/05_$1/ndmpcopy_status TSLOG=${LOG}_$timestamp src_filer='svlinf05' src_account='ndmp' src_passwd='src_passwd' dst_svm='svlinfsrc' dst_account='vsadmin-backup' dst_passwd='dst_passwd' host=`hostname` echo $host ssh -l root $src_filer "priv set -q diag ; ndmpcopy -sa $src_account:$src_passwd -da $dst_account:$dst_passwd -i $src_filer.eng.netapp.com:/vol/$1 10.56.10.161:/$dst_svm/$1" | tee -a $TSLOG echo "ndmpcopy Completed: `date` " } export -f ndmpcopy_cron_parallel /u/jsung/bin/parallel -j 0 --wd . --env ndmpcopy_cron_parallel --eta ndmpcopy_cron_parallel ::: local 

But the script crashed and complained that the exported ndmpcopy_cron_parallel function was not found:

 $ crontab -l 40 0,2,4,6,8,10,12,14,16,18,20,22 * * * /u/jsung/bin/ndmpcopy_cron_parallel_svlinf05.bash 

Error:

 Subject: Cron < jsung@cycrh6svl18 > /u/jsung/bin/ndmpcopy_cron_parallel_svlinf05.bash Computers / CPU cores / Max jobs to run 1:local / 2 / 1 Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete ETA: 0s Left: 1 AVG: 0.00s local:1/0/100%/0.0s **/bin/bash: ndmpcopy_cron_parallel: command not found** ETA: 0s Left: 0 AVG: 0.00s local:0/1/100%/0.0s 

They searched me and tried different things for a while. I even tweaked $ PATH. Not sure what I missed. Can we embed GNU Parallel in a BASH script and generally introduce crontab?

Answer 1

Congratulations. You were shocked by the shell .

There are two versions of bash installed on your system:

• / bin / bash v4.1.2. Old failed bash connection
• / usr / software / bin / bash v4.2.53 Average bash age fixed against Shellshock

The last number in the triple version of bash is the fix level. Shellshock error included several patches, but the corresponding one was 4.1.14, 4.2.50 and 4.3.27. This patch changes the format of the exported functions, as a result of which:

• If you export a function from pre-shellshock bash to post-shellshock bash, you will see a warning and the exported function will be rejected.
• If you export a function from post-shellshock bash to pre-shellshock bash, the export format of the function will not be recognized, so it will be ignored.

In both cases, the function will not be exported. In other words, you can only export a function between two versions of bash if both of them were fixed by shellshock, or if they were not fixed by shellshock.

Your script clearly indicates which bash to use to run it: one in / usr / software / bin / bash that has been fixed. The script calls the GNU parallel, and the GNU parallel then runs one or more subshells to run the commands. GNU parallel uses the value of the SHELL environment variable to find the shell that it should use.

I assume that your SHELL shell is set to /usr/software/bin/bash , and the environment where cron is running is set to /bin/bash . In this case, you will not have problems exporting the function when you try it from the bash prompt, but in the cron environment you will try to export the function from post-shellshock bash to pre-shellshock bash, and as described above, the result is that the export is silent ignored. Hence the error.

To work around this problem, you need to make sure that you are using the bash used to run the script command, the same as the bash used by GNU parallel. You could, for example, explicitly set the shell before invoking parallel GNU.

 export SHELL=/usr/software/bin/bash # ... /u/jsung/bin/parallel -j 0 --wd . --env ndmpcopy_cron_parallel --eta ndmpcopy_cron_parallel ::: local 

Or you can simply set it for the most parallel command:

 SHELL=/usr/software/bin/bash /u/jsung/bin/parallel -j 0 --wd . --env ndmpcopy_cron_parallel --eta ndmpcopy_cron_parallel ::: local 

Answer 2

As Richie says, the problem is most likely related to shellshock. Shellshock did not affect GNU Parallel, but fixes to fix shellshock disrupted function transfer using --env.

GNU Parallel catches up with the shellshock patch in Bash: Bash used BASH_FUNC_myfunc() as the variable name for exporting functions, but BASH_FUNC_myfunc%% used in later versions. Therefore, GNU Parallel needs to know this when passing a function.

The '()' version has been fixed in 20141022, and the '%%' version is expected to be fixed in 20150122. They should work in any combination. Thus, your remote Bash should not be fixed in the same way as your local Bash: GNU Parallel will “do the right thing” and there is no need to change your own code.

You can freely test the git version in which both are fixed: git clone git: //git.savannah.gnu.org/parallel.git