airflow-kube-setup
airflow-kube-setup copied to clipboard
bhavaniravi
How to deploy airflow on Kubernetes
Airflow Kubernetes Setup
This setup is used and tested in airflow 1.10.10. It's a good idea to use latest version of Airflow 2.3.x+. Need help setting up infrastructure and training your team? Contact me [email protected]
The setup files are copied directly from airflow's repo and modified to fit the requirements.
One major change is instead of building the docker image from source, we use pip to install airflow
Create Docker Image
cd scripts/docker
docker build -t airflow .
Push the image to your favorite registry and get the URL
Deploy in kubernetes
- Find $IMAGE in the repository
- Find $TAG in the repository
- Change it with the docker image URL and tag respectively
cd scripts/
./kube/deploy.sh -d persistent_mode
The other supported modes is git_mode
bhavaniravi