I want to add another DAG to an existing Airflow server. The server currently uses the LocalExecutor, but I may need my DAG to use CeleryExecutor. It seems that the configuration file airflow.cfgallows only one executor:
airflow.cfg
# The executor class that airflow should use. Choices include # SequentialExecutor, LocalExecutor, CeleryExecutor executor = LocalExecutor
Is it possible to configure Airflow so that existing DAGs can continue to use LocalExecutor, and my new DAG can use CeleryExecutor or its own artist class? I did not find examples of people who did this, and did not come across anything in the Airflow documentation.
It seems that the scheduler will only start one instance of the executor.
SubDAG , SubDagOperator. , SequentialExecutor:
bar_subdag = SubDagOperator( task_id='bar', subdag=my_subdag('foo', 'bar', default_args), default_args=default_args, dag=foo_dag, executor=SequentialExecutor() )
1.8, , 1.9 .
Source: https://habr.com/ru/post/1650064/More articles:Closing Elixir Phoenix: Failed to start baby - erlangIs it private necessary for a standalone Java application? - javaHow to update spark configuration after resizing work nodes in Cloud Dataproc - google-compute-engineThe most efficient way to split a python list into sublists using a key while maintaining order - pythonGolang for json to allow a value to not necessarily be an array - jsonAWS "eb deploy" always zips all files - gitSearch for the most frequently used word in a string field in the entire collection - stringThe real-time workflow when creating records is performed before the record is transferred to the database - c #Нет модуля с именем 'django.core.context_processors', в views.py - pythonAre empty curly brackets if the statements are satisfied? - phpAll Articles