airflow-chart
airflow-chart copied to clipboard
astronomer
KEDA v2 Supports StatefulSet
KEDA v2 supports not only Deployment but also StatefulSet. PR #731 Currently, this chart forces users to use deployment for Airflow workers instead of statefulsets. Is there any plan on auto-scaling Airflow workers as statefulsets via KEDA v2? If not can I contribute to this Chart?
Hey @jacobhjkim, just curious, what would be your use-case for stateful KEDA workers?
We also just merged a slightly more open-source friendly version of this chart into the official Apache Airflow repository, so a PR over on the main Airflow repo would be great! It lives under the chart subdirectory.
I currently have an issue with webserver not being able to fetch logs from each worker. I thought that this issue would be solved by using StatefulSet and assigning a fixed DNS name to each individual worker.
It's great to hear that this chart repo has been merged into the official Apache Airflow repo!
Oh, that's a good point. Even with a StatefulSet, if you attempt to view the logs from a worker that has been scaled away, you'd get he same error.
I think the only way to get around this would be to use a centralized logging solution, like one of the object store backends (s3, gcs, wasb) or elastic.
@jacobhjkim That's an interesting point re: logs. How would this work if a worker is scaled down? Would a worker need to spin up again to use that volume to pull the logs?=
Right, using StatefulSet won't solve the issue mentioned above. Will close this issue. (Sorry for the late reply 😅)