Removing Flow Scheduler Logs

Question

I am using Docker Apache airflow VERSION 1.9.0-2 ( https://github.com/puckel/docker-airflow ).

The scheduler generates a significant amount of logs, and the file system runs out of space quickly, so I'm trying to programmatically delete the scheduler logs created with airflow that are in the scheduler container in (/ usr / local / airflow / logs / scheduler))

I have all these maintenance tasks: https://github.com/teamclairvoyant/airflow-maintenance-dags

However, these tasks only delete the logs on the worker, and the scheduler logs are in the scheduler container.

I also set up remote logging by sending logs to S3, but as mentioned in this SO post. Deleting Airflow task logs, this setting does not stop recording airflow to the local computer.

In addition, I also tried to create a shared named volume between the worker and the scheduler, as described here: Docker Compose - split the named volume between multiple containers . However, I get the following error in an employee:

ValueError: Unable to configure handler 'file.processor': [Errno 13] Permission denied: '/usr/local/airflow/logs/scheduler'

and the following error in the scheduler:

ValueError: Unable to configure handler 'file.processor': [Errno 13] Permission denied: '/usr/local/airflow/logs/scheduler/2018-04-11'

And so, how do people delete the scheduler logs ??

Answer 1

Below is one of the solutions to this problem.

,

#>docker exec -it <name-or-id-of-container> sh

, .

cron rm .

Answer 2

, airflow-log-cleanup.py DAG ( ) , , .

, , , EC2 (7,7 /dev/xvda1), 30 DEFAULT_MAX_LOG_AGE_IN_DAYS , ( 4 DAG), 14 , :

DEFAULT_MAX_LOG_AGE_IN_DAYS = Variable.get("max_log_age_in_days", 30) DEFAULT_MAX_LOG_AGE_IN_DAYS = Variable.get("max_log_age_in_days", 14)

Answer 3

" Airflow" Airflow 1.10.

, Airflow (. UPDATING.md, README !!, Airflow source).

One caveat: because of the way logging, multiprocessing, and the default Airflow handlers interact, it is safer to override handler methods than extend them by calling super () in the handler derived class. Because Airflow handlers do not use locks by default