Multi controller support

[whynowy]

Summary

Similar to Argo Workflows, support multi controller deployment in one cluster/namespace.

Potential solution:

a.) Controller will have a instance-id configuration b.) Controllers having a instance-id will only take care of Pipeline and Vertex which has same instance-id annotated c.) Controller without instance-id config will not handle any pipeline, Vertex with different instance-id annotated d.) This logic applies to both cluster scope installation and namespace scope installation e.) How to handle InterStepBufferService? Shall we follow the same logic? f.) When doing multi-controller deployment in one namespace (either cluster scope or namespace scope), the configmaps used for different controller deployments should be different, and the admin (anyone who makes the deployment) is responsible for make the change in the deployment manifests, it is not required as a feature to be supported automatically without any manual change.

Use Cases

This can help Numaplane to do no-downtime upgrade when there's a controller upgrade.

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.