Config validation when dynamically adding and removing clusters.

[SamBarker]

When running kroxylicious as an ingress mechanism for a set of Virtual Clusters we expect the configuration to evolve (new clusters added, old clusters removed etc) ideally we want to do that seamlessly. However we do not expect our users to be perfect and get the configuration correct first time. Currently our strategy for validating cluster configuration is to throw exceptions and prevent the entre configuration model loading. That's fine for a stand alone proxy (i.e. in development or as a side car deployment) however it doesn't support our goal of seamless topology changes.

We need to revisit the way we handle configuration loading to ensure that valid clusters are exposed by kroxylicious even when there are errors in other cluster definitions.

This issue was inspired by some comments on a pull request:

I'm questioning the use of exceptions for configuration validation, at least in the medium term, as they make the configuration validation a binary decision. Where a single miss configured virtual cluster prevents the entire proxy from operating.

I'm starting to think the configuration should be loaded via a chain of validators any of which can prevent a virtual cluster from passing on. Only clusters which pass validation are actually loaded

Originally posted by @SamBarker in https://github.com/kroxylicious/kroxylicious/pull/290#discussion_r1194425032

Also

Yes, I was having thoughts along similar lines. In a world where virtual clusters can be added dynamically, we'd need a mechanism where the virtual cluster could 'fail' owning to a port conflict with those that exist already.

_Originally posted by @k-wall in https://github.com/kroxylicious/kroxylicious/pull/290#discussion_r1195467480

[k-wall]

This issue might be better rolled into the other issues we have discussing dynamic config reloading.