clickhouse operator fails to update pod labels on existing clichouse-keeper cluster

[mandreasik]

let's have this simple example where we set custom label app: custom-label for our cluster

---
apiVersion: clickhouse-keeper.altinity.com/v1
kind: ClickHouseKeeperInstallation
metadata:
  name: labels
  namespace: default
spec:
  configuration:
    clusters:
      - name: keeper
        layout:
          replicasCount: 3
  templates:
    podTemplates:
      - name: default
        metadata:
          labels:
            app: custom-label

on first apply cluster will be created, pods will have proper label.

now let's edit our manifest and change custom-label to custom-label2

after apply you will get following errors in operator log (operator used 0.23.5)

I0522 16:18:27.678358 1 reconciler.go:244] Updating existing ConfigMap
I0522 16:18:27.681495 1 reconciler.go:244] Updating existing StatefulSet
E0522 16:18:27.686147 1 reconciler.go:103] Error during reconcile. f: github.com/altinity/clickhouse-operator/pkg/controller/chk.(*ChkReconciler).reconcileStatefulSet-fm err: StatefulSet.apps "labels" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"app":"custom-label2"}: `selector` does not match template `labels`
2024-05-22T16:18:27Z ERROR Reconciler error {"controller": "clickhousekeeperinstallation", "controllerGroup": "clickhouse-keeper.altinity.com", "controllerKind": "ClickHouseKeeperInstallation", "ClickHouseKeeperInstallation": {"name":"labels","namespace":"default"}, "namespace": "default", "name": "labels", "reconcileID": "8c76066a-4831-4af9-a471-024a08bdc53d", "error": "StatefulSet.apps \"labels\" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{\"app\":\"custom-label2\"}: `selector` does not match template `labels`"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/clickhouse-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/clickhouse-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/clickhouse-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226
I0522 16:18:27.725570 1 reconciler.go:244] Updating existing ConfigMap
I0522 16:18:27.728170 1 reconciler.go:244] Updating existing StatefulSet