Create new Doc for "Running KubernetesPodOperator Tasks on External Cluster"

[paolaperaza]

Issue Type

Missing content for an existing feature

Links to Affected Docs

https://www.astronomer.io/docs/astro/kubernetespodoperator

The Issue or Context

We have a single KubernetesPodOperator doc for Astro, but there's a prominent use case that's only partially covered and we think warrants a dedicated doc: Running tasks with the KPO but on an external Kubernetes cluster instead of in Astro.

Customers might want to do this if:

• Security reasons - they want workers to run tasks in their network or on prem
• Customers want to use or access node types (e.g. GPUs) that Astronomer doesn't have available

Currently:

• There are technical discrepancies with current instructions
• There is little context around this use case and when it's recommended
• In @collinmcnulty's words:

The process is complex, and linking off to lots of other pages that aren't written with external KPO in mind complicates the comprehension. I think we could also add the errors you get if you skip each step, as its very easy to do and the errors are not intuitive.

Fix or Proposed Change

We should:

• Create a new dedicated doc for this
• Add more context around use case and when/why
• Review and retest procedure steps

Additional Notes

Slack thread -- https://astronomer.slack.com/archives/C015V2JFKT5/p1718160647008199

Our KPO doc for launching a pod in an external cluster lists having a network connection as a pre-requisite but doesn't mention you also need to install the aws cli in your docker image or otherwise you'll get:

the kubeconfig sample's whitespace is all wrong

We could maybe replace AWS CLI install with a python command using boto somehow? Adding the aws cli significantly increased the size of the image and the build time

Related:

• https://github.com/astronomer/docs/pull/3855

Required Reviewers

Collin McNulty, Tamara, Santiago