docker-airflow icon indicating copy to clipboard operation
docker-airflow copied to clipboard

Airflow 2 support

Open dataops-sre opened this issue 4 years ago • 0 comments

TL;TR

This repo seems no more maintained, I forked the original repo for airflow 2 support, check here. do: docker pull dataopssre/docker-airflow2:2.1.2

What does this PR do?

Add airflow2 support

Motivation

Airflow is been updated to version 2 and release its official docker image, you can also find bitnami airflow image. Nevertheless, puckel's image is still interesting, in the market none of providers offer an Airflow run with LocalExecutor with scheduler in one container, it is extremely usefull when to deploy a simple Airflow to an AWS EKS cluster. With Kubernetes you can resolve Airflow scablity issue by using uniquely KubernetesPodOpetertor in your dags, then we need zero computational power for airflow, it serves pure purpose of scheduler, seperate scheduler and webserver into two different pods is a bit problematic on AWS EKS cluster, we want to keep dags and logs into a Persistant volume, but AWS has some limitation for EBS volume multi attach, which means webserver and scheduler pod has to be scheduled on the same EKS node, it is a bit annoying. Thus puckel's airflow startup script is usefull.

Additional Notes

what this fork do :

  • Disactive by default the login screen in Airflow 2
  • Improve current script to only take into account Airflow environment variables
  • Make sure docker compose files works

dataops-sre avatar Aug 09 '21 11:08 dataops-sre