public-datasets-pipelines icon indicating copy to clipboard operation
public-datasets-pipelines copied to clipboard

Consider utilizing GKE Operators in anticipating of shifting to Composer v2

Open leahecole opened this issue 4 years ago • 1 comments

Composer v2 was released this week, and it uses GKE Autopilot under the hood. With GKE Autopilot, we can no longer use node pool affinity with the KubernetesPodOperator. In anticipation of eventually migrating to Composer v2 we can utilize the GKEStartPodOperator, a child of the KubernetesPodOperator in one of two ways:

  1. Create a long lasting GKE Cluster with node pools that is managed outside of Airflow, use the GKEStartPodOperator with node pool affinity as needed.
  2. Programmatically create an ephemeral GKE Cluster with node pools using the GKECreateClusterOperator, use the GKEStartPodOperator to launch pods with node pool affinity, and tear down the cluster with the GKEDeleteClusterOperator

leahecole avatar Sep 16 '21 19:09 leahecole

This was brought up as a discussion regarding #180

leahecole avatar Sep 16 '21 19:09 leahecole