Support for Passing Data among Argo Jobs

[kaushikb11]

Feature Request

Let’s consider I have triggered an Argo Workflow, it is a DAG of multiple Jobs. Jobs A and B are running parallelly & we want data to pass from Job A to Job B. Example: Job B does Tracking & Job A does OD with Bounding Boxes as outputs. Right now on our platform, we pass data through queues and let the Jobs polls on it.

It would help in extending Onepanel’s Workflow (Argo) capability for building Video Applications or Batch Processing Jobs.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label kind/enhancement to this issue, with a confidence of 0.85. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[rushtehrani]

Related Argo issues https://github.com/argoproj/argo-workflows/issues/2729 https://github.com/argoproj/argo-workflows/issues/2551

[sarabala1979]

@rushtehrani are you using Argo workflow? would you like to add onepanel in argo users.md?

[sarabala1979]

I would like to here the more details for the Stream workflow usecase. Basically, I am waiting for real-world usecases to start the workflow.

[rushtehrani]

are you using Argo workflow? would you like to add onepanel in argo users.md?

@sarabala1979 Yes we are and I will absolutely send a PR to add us to users.md.

I would like to here the more details for the Stream workflow usecase. Basically, I am waiting for real-world usecases to start the workflow.

@kaushikb11 has a great use case with videos. It's basically a combination of this issue and https://github.com/onepanelio/core/issues/613.

I am also exploring other related use cases and will provide more information in https://github.com/argoproj/argo/issues/2729 based on my findings.