Taleb Zeghmi
Taleb Zeghmi
Why not compile to Kubeflow Pipelines via an intermediate representation (IR) [1]? [1] https://github.com/kubeflow/pipelines/issues/3703
Ideally the user could set a TTL rather than managing a job to GC...
another mention Spark Pandas https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/index.html
see: https://registry.opendata.aws/
If using https://github.com/zillow/intake-nested-yaml-catalog then you can define it as such getting free tab completion and the Intake Gui.  ```yaml metadata: hierarchical_catalog: true population: asia: china: driver: csv args: url:...
I'd be happy to contribute on how we use Intake at Zillow too!
> The benefit of putting it in hvPlot from an intake user's perspective would be that you could declare it in the catalog yaml. Another benefit is that hvPlot also...
We at a Zillow use [synchronization-wf-level.yaml](https://github.com/argoproj/argo-workflows/blob/main/examples/synchronization-wf-level.yaml) workflow semaphore to limit concurrency at the WorkflowTemplate level.
related https://github.com/argoproj/argo-events/issues/1111
I'm hitting the same issue, when running [Metaflow](https://github.com/netflix/metaflow/) on Argo with a `join` step. There is no simple workaround because we'd have to recompile all customer Metaflow flows to prevent...