Problems with the airflow scheduler as a daemon process

I have an EC2 instance that works with 1.8.0 airflow using LocalExecutor. In the docs, I expected one of the following two commands to raise the scheduler in daemon mode:

airflow scheduler --daemon --num_runs=20

or

airflow scheduler --daemon=True --num_runs=5

But this is not so. The first command seems to work, but it just returns the following output before returning to the terminal without creating a background task:

[2017-09-28 18:15:02,794] {__init__.py:57} INFO - Using executor LocalExecutor
[2017-09-28 18:15:03,064] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python3.5/lib2to3/Grammar.txt
[2017-09-28 18:15:03,203] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python3.5/lib2to3/PatternGrammar.txt

The second command throws an error:

airflow scheduler: error: argument -D/--daemon: ignored explicit argument 'True'

This is odd because, according to docs, it --daemon=True should be a valid argument to call airflow scheduler.

, qaru.site/questions/1686582/..., systemd , .

script . m4.xlarge EC2 Ubuntu 16.04.3:

:

sudo systemctl enable airflow-scheduler
sudo systemctl start airflow-scheduler

. DAG, , , , , .

journalctl -f . . , , DAG , . -?

Sep 28 18:39:30 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:30,965] {dag_processing.py:627} INFO - Started a process (PID: 21822) to generate tasks for /home/ubuntu/airflow/dags/scheduler_test_dag.py - logging into /home/ubuntu/airflow/logs/scheduler/2017-09-28/scheduler_test_dag.py.log
Sep 28 18:39:31 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:31,016] {jobs.py:1002} INFO - No tasks to send to the executor
Sep 28 18:39:31 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:31,020] {jobs.py:1440} INFO - Heartbeating the executor
Sep 28 18:39:32 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:32,022] {jobs.py:1404} INFO - Heartbeating the process manager
Sep 28 18:39:32 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:32,023] {jobs.py:1440} INFO - Heartbeating the executor
Sep 28 18:39:33 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:33,024] {jobs.py:1404} INFO - Heartbeating the process manager
Sep 28 18:39:33 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:33,025] {dag_processing.py:559} INFO - Processor for /home/ubuntu/airflow/dags/capone_dash_dag.py finished
Sep 28 18:39:33 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:33,026] {dag_processing.py:559} INFO - Processor for /home/ubuntu/airflow/dags/scheduler_test_dag.py finished

airflow scheduler , . DAG 9 , , . , systemd , , .

?

+4
2

?

Airflow

airflow kerberos -D
airflow scheduler -D
airflow webserver -D

airflow webeserver --help ( 1.8):

-D, --daemon Daemonize

, . .

airflow scheduler -D:

, , , . , $AIRFLOW_HOME/airflow-scheduler.pid. , $AIRFLOW_HOME/airflow-scheduler.err, , lockfile.AlreadyLocked: /home/ubuntu/airflow/airflow-scheduler.pid is already locked. , , :

sudo rm $AIRFLOW_HOME airflow-scheduler.err  airflow-scheduler.pid
airflow scheduler -D 

.

+3

systemd:

PATH , . , /etc/sysconfig/airflow:

PATH=/home/ubuntu/bin:/home/ubuntu/.local/bin:$PATH

:

PATH=/home/ubuntu/bin:/home/ubuntu/.local/bin

, PATH /bin, bash, LocalExecutor .

, AIRFLOW_HOME. , Airflow .

+2

Source: https://habr.com/ru/post/1686577/


All Articles