How to start a daily DAG launch at midnight local time, not midnight UTC

Question

I am in the UTC + 4 time zone, so when Airflow launches overnight ETLs, it is already 4:00 here. How can I say that Airflow starts the launch for day ds already on day ds-1 at 20:00, but with ds = ds?

It is strongly recommended that documents store all servers in UTC, so I am looking for a solution at the application level.

EDIT: The hacker solution is to define it so that it works every day at 20:00, so on the “previous” day, but then use tomorrow_ds instead of ds in the job. But it still looks weird in the Airflow user interface because it will show the UTC runtime.

Answer 1

The schedule interval can also be a “cron expression”, which means you can easily start it at 20:00 UTC. This, combined with "user_defined_filters", means that you can get the desired behavior with a little cheating:

 from airflow.models import DAG from airflow.operators.bash_operator import BashOperator from datetime import datetime import pytz tz = pytz.timezone('Asia/Dubai') def localize_utc_tz(d): return tz.fromutc(d) default_args = { 'start_date': datetime(2017, 11, 8), } dag = DAG( 'plus_4_utc', default_args=default_args, schedule_interval='0 20 * * *', user_defined_filters={ 'localtz': localize_utc_tz, }, ) task = BashOperator( task_id='task_for_testing_file_log_handler', dag=dag, bash_command='echo UTC {{ ts }}, Local {{ execution_date | localtz }} next {{ next_execution_date | localtz }}', ) 

It is output:

UTC 2017-11-08T20: 00: 00, local 2017-11-09 00: 00: 00 + 04: 00 next 2017-11-10 00: 00: 00 + 04: 00

You need to be careful about the "types" of the variables you use. For example, ds and ts are strings, not datetime objects, which means the filter will not work on them

Answer 2

Can you write a python utility that rewrites your tz chart in UTC? https://github.com/bloomberg/tzcron/blob/master/tzcron.py

EDIT: There is a recent commit that tells Airzone Timezone: https://github.com/apache/incubator-airflow/commit/f1ab56cc6ad3b9419af94aaa333661c105185883