Comparison with Helm's official chart and airflow-on-k8s-operator

[Minyus]

What are the differences with Helm's official chart and apache/airflow-on-k8s-operator in terms of the architecture/design? What are the pros and cons other than helm support?

[schnie]

Hey @Minyus, the chart in the helm repo was maintained by some open source developers. To avoid this confusion going forward, we've actually donated this chart to the official Apache Airflow repo, so that the project can actually have an "official" helm chart. This PR was just merged 6 days ago (https://github.com/apache/airflow/pull/8777), so there's still some work to do to get it published. Ideally we can all work on the same chart over here, instead of having multiple Airflow charts out in the wild.

As for the operator, I don't know anyone using it, but it takes a completely different approach to deploying Airflow. I believe it was developed for the GKE marketplace. Not sure if anyone is using it in production or not.

[Minyus]

Thanks for your reply, @schnie .

So there are at least 3 options:

• Official Helm chart for Apache Airflow maintained by Helm
• Official Helm chart for Apache Airflow maintained by Apache Airflow (formerly Astronomer)
• Airflow on K8s Operator maintained by Apache (formerly GoogleCloudPlatform)

Do you have any ideas about the pros and cons of Apache Airflow's (formerly Astronomer's) Helm chart?

[schnie]

I think if you plan to bake your DAG files in your code and are using postgres as a database, then the Apache/Astronomer should be a pretty solid choice (more PRs will open up new options). It includes other things like prometheus metrics integration, migration control, and a bunch of other production-ready things.

What does your environment look like?

[Minyus]

Thank you for your reply.

We are exploring a good way to run Airflow operations in Kubernetes.

It seems a major difference is the way to update DAGs.

According to https://github.com/helm/charts/tree/master/stable/airflow#option-2---mount-a-shared-persistent-volume , Official Helm chart by Helm can update DAGs without building a new Airflow Docker image, but I couldn't find the option in Apache/Astronomer's Helm Chart.

I filed a new issue "How to update DAGs without building a new Airflow Docker image?" at: https://github.com/astronomer/airflow-chart/issues/94

[ecerulm]

we've actually donated this chart to the official Apache Airflow repo, so that the project can actually have an "official" helm chart. Ideally we can all work on the same chart over here, instead of having multiple Airflow charts out in the wild.

@schnie, so it's the idea that this repo contains the "latest" and it will be "copied" regularly to https://github.com/apache/airflow/tree/master/chart?

Or will they diverge?

[schnie]

@ecerulm before donating the helm chart to the Apache Airflow repo, we made some changes to the code to scrub anything astronomer related. That change broke some functionality between our helm chart and our platform, so this repo exists for our platforms usage.

Our goal is to actually completely shut this repo down and utilize the open source version of the chart, but we have some work to do still. The two charts have diverged a little bit, but we aim to get synced up very soon.

[danielhoherd]

This chart now uses the apache airflow chart as a dependency, so I think the original ask in the ticket is no longer relevant. Closing.