Multi controller support

[whynowy]

Summary

Similar to Argo Workflows, support multi controller deployment in one cluster/namespace.

Potential solution:

a.) Controller will have a instance-id configuration b.) Controllers having a instance-id will only take care of Pipeline and Vertex which has same instance-id annotated c.) Controller without instance-id config will not handle any pipeline, Vertex with different instance-id annotated d.) This logic applies to both cluster scope installation and namespace scope installation e.) How to handle InterStepBufferService? Shall we follow the same logic? f.) When doing multi-controller deployment in one namespace (either cluster scope or namespace scope), the configmaps used for different controller deployments should be different, and the admin (anyone who makes the deployment) is responsible for make the change in the deployment manifests, it is not required as a feature to be supported automatically without any manual change.

Use Cases

This can help Numaplane to do no-downtime upgrade when there's a controller upgrade.

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[vigith]

should instance-id be the numaflow version?

[whynowy]

should instance-id be the numaflow version?

It should not be tied to version, the admin (or any platform automates this) should decide what to use. Nothing stops ppl running multiple controllers for different namespaces in one cluster, even they are in the same version.

[kohlisid]

So in a single namespace as well we can have multiple controllers running the same version?

[whynowy]

So in a single namespace as well we can have multiple controllers running the same version?

There's no need for namespaces scope installation, but it should be doable if anyone wants to do it.