docker-airflow
docker-airflow copied to clipboard
Airflow 2 support
TL;TR
This repo seems no more maintained, I forked the original repo for airflow 2 support, check here. do:
docker pull dataopssre/docker-airflow2:2.1.2
What does this PR do?
Add airflow2 support
Motivation
Airflow is been updated to version 2 and release its official docker image, you can also find bitnami airflow image. Nevertheless, puckel's image is still interesting, in the market none of providers offer an Airflow run with LocalExecutor with scheduler in one container, it is extremely usefull when to deploy a simple Airflow to an AWS EKS cluster. With Kubernetes you can resolve Airflow scablity issue by using uniquely KubernetesPodOpetertor in your dags, then we need zero computational power for airflow, it serves pure purpose of scheduler, seperate scheduler and webserver into two different pods is a bit problematic on AWS EKS cluster, we want to keep dags and logs into a Persistant volume, but AWS has some limitation for EBS volume multi attach, which means webserver and scheduler pod has to be scheduled on the same EKS node, it is a bit annoying. Thus puckel's airflow startup script is usefull.
Additional Notes
what this fork do :
- Disactive by default the login screen in Airflow 2
- Improve current script to only take into account Airflow environment variables
- Make sure docker compose files works