How to get and process mysql records using Airflow?

Question

I need

1. run a select query on MYSQL DB and fetch the records. 2. Records are processed by python script. 

I am not sure how I should act. Is xcom the way here? In addition, MYSQLOperator only executes a query, does not retrieve records. Is there a built-in transfer operator that I can use? How can I use MYSQL here?

you can use PythonOperator, which uses a hook to get the data, apply the conversion and send the (now typed) lines back to another place.

Can someone explain how to proceed regarding the same.

Refer - http://markmail.org/message/x6nfeo6zhjfeakfe

 def do_work(): mysqlserver = MySqlHook(connection_id) sql = "SELECT * from table where col > 100 " row_count = mysqlserver.get_records(sql, schema='testdb') print row_count[0][0] callMYSQLHook = PythonOperator( task_id='fetch_from_testdb', python_callable=mysqlHook, dag=dag ) 

Is it correct? Also, how do we use xcoms to store records for the next MySqlOperator?

 t = MySqlOperator( conn_id='mysql_default', task_id='basic_mysql', sql="SELECT count(*) from table1 where id > 10", dag=dag) 

Answer 1

Of course, just create a trap or statement and call the get_records () method: https://airflow.readthedocs.io/en/stable/_modules/airflow/hooks/dbapi_hook.html