POD annotations are dropped with the reconcile of CHK STS

[jirislav]

Keeping the POD annotations is essential to run the workload in EKS, where the fargate profile is the default one.

Dropping essential annotations, such as "eks.amazonaws.com/compute-type" = "ec2" will cause the POD to be unschedulable due to the fact that:

• you can't mount a volume to a fargate node.
• you can't possibly satisfy the nodeSelector & nodeAffinity rules with fargate profile in case you ask for dedicated EC2 node for billing purposes.

Example manifest:

apiVersion: "clickhouse-keeper.altinity.com/v1"
kind: "ClickHouseKeeperInstallation"
metadata:
  name: clickhouse-keeper
spec:
  configuration:
    clusters:
      - name: chk
        layout:
          replicasCount: 3
  templates:
    podTemplates:
      - name: clickhouse-keeper
        metadata:
          annotations:
            eks.amazonaws.com/compute-type: "ec2"

Interestingly, first POD of the 3 replicas starts with correct annotation, but then, the second doesn't as the annotations are dropped from the underlying statefulset.

Note that I also see this in the log of the operator, which is possibly the result of this behavior:

E0802 07:02:22.310975       1 reconciler.go:299] err: Operation cannot be fulfilled on clickhousekeeperinstallations.clickhouse-keeper.altinity.com "chk": the object has been modified; please apply your changes to the latest version and try again