How to run bash script file in Airflow

Question

I have a bash script that creates a file (if it does not exist) that I want to run in Airflow, but when I tried it. How to do it?

#!/bin/bash #create_file.sh file=filename.txt if [ ! -e "$file" ] ; then touch "$file" fi if [ ! -w "$file" ] ; then echo cannot write to $file exit 1 fi 

and here is what I call it in Airflow:

 create_command = """ ./scripts/create_file.sh """ t1 = BashOperator( task_id= 'create_file', bash_command=create_command, dag=dag ) lib/python2.7/site-packages/airflow/operators/bash_operator.py", line 83, in execute raise AirflowException("Bash command failed") airflow.exceptions.AirflowException: Bash command failed 

Answer 1

From the tutorial, this is normal:

 t2 = BashOperator( task_id='sleep', bash_command='sleep 5', retries=3, dag=dag) 

But you pass a multiline command

 create_command = """ ./scripts/create_file.sh """ 

should be

 create_command = "./scripts/create_file.sh " 

In addition, you must also make sure that you are in the correct directory to avoid cryptic errors. Do this, for example, like this:

 create_command = "./scripts/create_file.sh" if os.path.exists(create_command): t1 = BashOperator( task_id= 'create_file', bash_command=create_command, dag=dag ) else: raise Exception("Cannot locate {}".format(create_command))