conduit icon indicating copy to clipboard operation
conduit copied to clipboard

Published 20 hours ago • verified publisher icon ConduitIO

Clustered Conduit

Open neovintage opened this issue 3 years ago • 1 comments

The problem to solve is High Availability. Should one node go down, another will need to take it's place. The goal is not disaster recovery. We will need to develop a separate solution for that.

neovintage avatar May 03 '21 20:05 neovintage

Random thought about this: we should think about how to work with standalone plugins when Conduit is clustered. The plugin executables need to be found on all instances (if we want to make it possible that any Conduit instance in a cluster can work on any pipeline).

hariso avatar Mar 25 '22 15:03 hariso

I think this is still pretty far out in the future @neovintage? Especially if there's a conduit Kubernetes operator which can help with orchestration.

simonl2002 avatar Apr 06 '23 15:04 simonl2002

Closing, the Conduit Kubernetes operator work will address running Conduit in a distributed manner.

simonl2002 avatar Jul 26 '23 16:07 simonl2002