airflow-maintenance-dags
airflow-maintenance-dags copied to clipboard
[Airflow 2.0] Using log-cleanup in Docker
Hi, i wanna ask, why it doesn't work for my airflow on docker? The dags is ok but it won't delete my logs on local and so in the webserver it self
and this is modified log-cleanup.py
# # #32
the log shows this: and in my docker the log folder is ready: and I try to mount airflow.cfg into my docker, like this:
i think you need to set this Airflow Variable > 0: airflow_log_cleanup__max_log_age_in_days
https://github.com/teamclairvoyant/airflow-maintenance-dags/tree/master/log-cleanup#airflow-log-cleanuppy
airflow_log_cleanup__max_log_age_in_days - integer - Length to retain the log files if not already provided in the conf. If this is set to 30, the job will remove those files that are 30 days old or older.
i think you need to set this Airflow Variable > 0:
airflow_log_cleanup__max_log_age_in_days
https://github.com/teamclairvoyant/airflow-maintenance-dags/tree/master/log-cleanup#airflow-log-cleanuppy
airflow_log_cleanup__max_log_age_in_days - integer - Length to retain the log files if not already provided in the conf. If this is set to 30, the job will remove those files that are 30 days old or older.
i set to 0 because i want to delete the log immediately. But i think i know my issue that base_log_folder that connect with airflow.cfg but i don't know how to mount that on my docker
what docker image you use for this airflow?
if you use official one, you can add this as env in your docker-compose (since you already mounted it as ./logs:/opt/airflow/logs
)
AIRFLOW__LOGGING__BASE_LOG_FOLDER: '/opt/airflow/logs'
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#base-log-folder
what docker image you use for this airflow?
if you use official one, you can add this as env in your docker-compose (since you already mounted it as
./logs:/opt/airflow/logs
)
AIRFLOW__LOGGING__BASE_LOG_FOLDER: '/opt/airflow/logs'
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#base-log-folder
I use this airflow 2.0.1 this is my docker images list
Thank you @Alionar for your response, I'm glad with this community who help newbie just like me. I'm sorry if my english is not good enough to explain my problem. Ok now I'm gonna tell my problem is, I need a solution for my airflow-log cleanup and email opearator. I've try with another airflow os (airflow ova via oracle vm) that include airflow.cfg and it's work normaly. but when i try to run my dags with docker composer, there're no task done with my dags (the cleanup and email operator not work). I think this is due to the airflow cfg not working with docker-composer properly. I hope you don't mind with my question, and thank you for your explaination.
I'm also experiencing the same issue with log cleanup dag. I've already set the airflow_log_cleanup__max_log_age_in_days
to 3 (days) and airflow_log_cleanup__enable_delete_child_log
to True. But the dag didn't clean the logs on the airflow/logs/scheduler
directory
And the logs showed there's no data inside the scheduler