Changes to "immutable" fields of base cluster results in poor UX for existing clusters (OutOfSync)

[bcle]

When a user pushes a change to a base cluster that affects fields that CAPI providers treat as immutable, such as the instanceType or additionalTags of the AWSMachineTemplate spec, then:

1. New clusters will behave as expected, and acquire the settings of the modified base cluster
2. Unfortunately, existing clusters will be flagged as OutOfSync because some sub-resources (e.g. AWSMachineTemplate) will never reach their desired state, since the corresponding CAPI provider will reject the applied state, with an error indicating that the requested fields are immutable. Those clusters are still functional, but will look unhealthy to the ArgoCD user.

It is (2) that leads to a poor UX. Some possible work-arounds, none of which are appealing:

• Advise the user to only update mutable fields, such as kubernetesVersion and replicas. For immutable fields (our documentation would need to provide a guide or documentation), recommend making a new, separate copy of the base cluster, and clone new clusters from it, leaving existing clusters to use the old copy.
• Enhance arlon to give the option to add a ignoreDifferences section to the existing ArgoCD application for existing clusters, to ignore immutable fields.

Aha! Link: https://pf9.aha.io/features/ARLON-257