Flagger should support custom annotations on VirtualService/DestinationRule resources

[rolandkool]

Describe the feature

We have a requirement to add custom annotations to VirtualService and DestinationRule resources. Our use case is that we add an annotation to indicate the resource needs to be synced to other clusters as we run Istio in multi-primary mode.

If we add a custom annotation through a mutating webhook, Flagger ends up in a loop where it cannot progress because the mutation triggers a reconcile and the VirtualService is reset to a weight of 100/0.

Proposed solution

We'd like to have a possibility for Flagger to either ignore custom annotations (through a parameter that can be specified when start flagger) or a way to specify them through the canary custom resource, similar as to what can be done for the Service to support External DNS annotations.

We don't have a preference for either solution.

[djazayeri]

I have a related use case, which would be solved by one of the proposed solutions.

I depend on a different controller that will add a custom annotation on a Service resource in the cluster. I need flagger to ignore that particular annotation, and not reconcile it back. I do not own the other controller, and it's not possible to have it modify my flagger canary service, it only applies annotations directly in the cluster. I know the annotation key, but I don't know the value it will have (it will be generated by the other controller), so I cannot apply this in my canary resource.

Allowing Flagger to ignore certain annotations (by name) on a Service would solve this problem.