Configurable Ephemeral Storage for Worker Pods

[chomipi88]

New Feature Description

Today Astro Hosted Workers (Celery & Kubernetes) are both limited to 10Gi of ephemeral storage. Ephemeral storage refers to the filesystem that exists inside each worker. Any files created and written to the filesystem during the life of any given worker counts against this quota.

Goals

The primary goal of this project is to give customers access to much larger storage capacity in their Airflow workers, while billing them for the additional requested capacity. At a high level this includes:

1. Celery Executor Workers - Allow customers to configure a new property on their Celery Executor Worker Queues. This new field will be an integer representing the amount of ephemeral storage that we will request in our Kubernetes resources in Gibibytes.
2. Kubernetes Executor Workers - Allow customers to also specify ephemeral storage requests and limits in their DAG code, specifically the executor_config section on task definitions.
3. Both Workers - Allow customers to override the default ephemeral storage for KubernetesExecutor pods, as well as KubernetesPodOperators.

We’ll keep our 10Gi as the default and minimum, but allow customers to select higher values if they need additional capacity, up to a max of 100Gi.

Required Reviewers

chomipi88

Links to Internal Info or Resources

https://www.notion.so/astronomerio/Astro-Hosted-Ephemeral-Storage-d198aaff1b6c4d0288b9bb5dee33fa49

Release Date

Priv Preview end of Feb

Additional Notes

No response