Configure Airflow to work with CeleryExecutor

Question

I am trying to configure Airbnb AirFlow to use CeleryExecutor as follows:

I changed executer in airflow.cfg from SequentialExecutor to CeleryExecutor :

 # The executor class that airflow should use. Choices include # SequentialExecutor, LocalExecutor, CeleryExecutor executor = CeleryExecutor 

But I get the following error:

 airflow.configuration.AirflowConfigException: error: cannot use sqlite with the CeleryExecutor 

Note that sql_alchemy_conn configured as follows:

 sql_alchemy_conn = sqlite:////root/airflow/airflow.db 

I looked at Airflow GIT ( https://github.com/airbnb/airflow/blob/master/airflow/configuration.py )

and found that the following code throws this exception:

 def _validate(self): if ( self.get("core", "executor") != 'SequentialExecutor' and "sqlite" in self.get('core', 'sql_alchemy_conn')): raise AirflowConfigException("error: cannot use sqlite with the {}". format(self.get('core', 'executor'))) 

It appears from this validate method that sql_alchemy_conn cannot contain sqlite .

Do you have an idea on how to configure CeleryExecutor without sqllite? note that I downloaded rabitMQ to work with CeleryExecuter as needed.

Answer 1

According to AirFlow, CeleryExecutor requires a different backend than the default SQLite database. For example, you should use MySQL or PostgreSQL .

sql_alchemy_conn in airflow.cfg needs to be changed to follow the SqlAlchemy connection string line (see SqlAlchemy Document )

For instance,

 sql_alchemy_conn = postgresql+psycopg2://airflow: airflow@127.0.0.1 :5432/airflow 

Answer 2

To configure Airflow for mysql first install mysql this may help or just google it

• goto airflow installation director usually / home // airflow
• edit airflow.cfg

• to find

  sql_alchemy_conn = sqlite: ////home/vipul/airflow/airflow.db

and add # in front of it so it looks like

 #sql_alchemy_conn = sqlite:////home/vipul/airflow/airflow.db 

if you have sqlite by default

• add this line below

  sql_alchemy_conn = mysql: //: @localhost: 3306 /

• save file

• execute command

  initdb airflow

and done!

Answer 3

As pointed out in other answers, you need to use a database other than SQLite. In addition, you need to install rabbitmq, configure it accordingly and modify each of your airflow.cfg to have the correct rabbit information. For an excellent guide on this, see the Airflow Server / Cluster Creation Guide .