old-argo-dataflow icon indicating copy to clipboard operation
old-argo-dataflow copied to clipboard

Published 20 hours ago • verified publisher icon argoproj-labs

ArgoLabs Dataflow is on hold

Open alexec opened this issue 3 years ago • 10 comments

We've put Dataflow on hold as we are planning on some significant changes and redesign. Stay tuned.

alexec avatar Jan 28 '22 00:01 alexec

@alexec we would love to start testing dataflow, do you maybe have some insights in what the changes will be? and how fundamental :dancers:

Bouwdie avatar Mar 09 '22 12:03 Bouwdie

We're currently working a new version:

  • It's focussed on providing ways to do windowing operations. This is something difficult for the old version.
  • It's currently supports Kafka, does not provide support for the variety of source and sinks.
  • It has many similar concepts and ideas as this version (e.g. pipelines are CRDs).
  • It should be an easy migration from the older version to this version.
  • It should be available in about 3mo.

With that in mind, any testing you do with old dataflow, will be extremely instructive and helpful.

alexec avatar Mar 09 '22 16:03 alexec

Already excited to see this unfold! Are you looking for contributors?

Also another question: Is there any aspiration to allow for batch jobs, ie reading a bounded source including fan-out and -in? Use case is finite data processing jobs, along ETL lines really.

cdeboeser avatar Mar 17 '22 10:03 cdeboeser

Hello, Why is it not built in the opensource, isn't argo a CNCF project?

ant31 avatar May 10 '22 13:05 ant31

Hey, is this still in the works? I'm wondering if Nats streaming will still be supported?

BenCoughlan15 avatar Jun 17 '22 09:06 BenCoughlan15

Hey, is this still in the works? When this can be used in production?

dafang avatar Jun 21 '22 02:06 dafang

Hey guys, any update on the re-launch date yet?

openrory avatar Aug 03 '22 14:08 openrory

Argo 3.4 rc1 has removed dataflow pipelines from the codebase

https://github.com/argoproj/argo-workflows/issues/8998

From earlier in the thread, the idea was that it should be possible to upgrade easily .

It should be an easy migration from the older version to this version.

Will that still be the case @alexec

louisnow avatar Sep 13 '22 11:09 louisnow

@edlee2121 can I please ask you to comment on this issue to give users insight into the future.

alexec avatar Sep 13 '22 17:09 alexec

Hi all, dataflow is being reworked as part of a new project focused on operational analytics. Please check it out. https://github.com/numaproj/numaflow

edlee2121 avatar Sep 13 '22 20:09 edlee2121

Now that we have open-sourced numaflow - https://github.com/numaproj/numaflow, shall we close this issue?

vigith avatar Nov 04 '22 22:11 vigith