[Discussion] Dynamic task generation

[yxue-kabam]

Description I have been working on a proposal to generate tasks dynamically for argo dsl.

I got to a point where I think I should get your feedback about whether it is a feasible design.

Currently, this is a work in progress, and I am still picking up the metaprogramming mechanics implemented in argo dsl. I wouldn't be surprised if my design isn't actually feasible and needed to be abandoned, but I did my best in terms of standing from a data science perspective that have authored airflow pipelines.

The proposal is here: https://gist.github.com/binarycrayon/75af90c1cdf660333f9903cd5822245d

Please let me know what you think!

Additional context This has originally sent as email, and forwarded here per Marek's suggestion. I realize that argo also have a built in with_items or with_params https://github.com/argoproj-labs/argo-client-python/search?q=with_items&unscoped_q=with_items

Technically I'm no longer blocked for what I need to do, but I still want to get feedback

[etheleon]

i cant get with_params working

[binarycrayon]

@etheleon https://github.com/argoproj-labs/argo-python-dsl/blob/master/tests/workflows/loops.py