soopervisor
soopervisor copied to clipboard
☁️ Export Ploomber pipelines to Kubernetes (Argo), Airflow, AWS Batch, SLURM, and Kubeflow.
We want to add support for pipelines exported to argo and kubeflow to request specific resources for a given task (memory, CPU, GPU). I'm thinking we could have a mapping...
one that does not use the sample slurm cluster
[the current example](https://soopervisor.readthedocs.io/en/latest/api/kubernetes.html) works for local cluster but won't work for a production one, it needs to be a PVC ([see an example here](https://soopervisor.readthedocs.io/en/latest/tutorials/kubernetes.html#google-cloud)) we need to ensure that this...
I tried running the example (using KubernetesPodOperator) and it failed, something in Airflow broke. I tried downgrading Airflow, the [k8s provider package](https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/), and the [k8s package](https://pypi.org/project/kubernetes/), but didn't find the...