Support workspace read (and write?) for loop items.

[wzhanw]

/kind feature

Description:

Add support for COS workspace read (and write?) for loop item, as lists will probably be considered artifacts most of the time. Allow to mark it somehow, e.g. as iterateParamIsArtefact: true.

Impact Without this, passing big outputs as artifacts won't work with loops, nor parallel for sequential.

[wzhanw]

@Tomcli @pugangxa could you take a loop at this requirement?

The loop items are read/write as artifacts from the cos bucket, we need to iterate on those artifacts and may pass the artifact to the sub pipeline inside the loop.

[Tomcli]

thanks @wzhanw. We added a way to pass down the workspace to the pipelineRuns inside loops https://github.com/kubeflow/kfp-tekton/pull/787

It makes sense to add a way to iterate items inside a file in the cos bucket. Argo has similar feature like withSequence, but expecting users to take the index number and extra the file inside their component logic. https://github.com/argoproj/argo-workflows/blob/master/examples/handle-large-output-results.yaml

[pugangxa]

Yes #787 together with #753 can meet the big data passing needs but still need logic to treat the lists as the artifacts to read.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.