Upgrading clickhouse-operator from 0.17 to 0.18 causes reconcilation

[tdminh]

Hi,

In order to resolve this error "ClickHouse Operator not able to watch ClickHouseOperatorConfiguration Resources", I have just tried to upgrade our clickhouse-operator to the newer version. Everything works perfectly but the upgrade causes reconcilation for some reasons. Does any body have the same problem?

I0131 08:36:41.871751       1 controller.go:416] Starting ClickHouseInstallation controller
I0131 08:36:41.871771       1 controller.go:848] waitForCacheSync():Syncing caches for ClickHouseInstallation controller
I0131 08:36:42.011840       1 controller.go:517] ENQUEUE new ReconcileCHI cmd=add for clickhouse/analytics
...
I0131 08:36:42.285821       1 controller.go:607] addChopConfig():kube-system/chop-config:already known config - do nothing
W0131 08:36:52.285254       1 normalizer.go:926] unable to parse secret field address:
...
I0131 08:36:52.380321       1 worker.go:410] markReconcileStart():clickhouse/analytics/91d89fc4-d403-4a4c-91b9-e89fca427ed1:reconcile started
...
W0131 08:36:52.450212       1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0131 08:36:52.450851       1 worker.go:1148] unable to create PDB poddisruptionbudgets.policy "analytics" already exists
...
I0131 08:36:52.646915       1 worker.go:870] reconcileHost():Reconcile Host 0-0 started
I0131 08:36:52.760510       1 creator.go:567] getPodTemplate():clickhouse/analytics/91d89fc4-d403-4a4c-91b9-e89fca427ed1:statefulSet chi-analytics-cluster-0-0 use custom template clickhouse:21.8.13.6-with-backup
...
I0131 08:36:52.764216       1 worker.go:1821] getStatefulSetStatus():INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-analytics-cluster-0-0
...
I0131 08:36:53.165439       1 worker.go:1821] getStatefulSetStatus():INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-analytics-cluster-0-0
I0131 08:36:53.165610       1 worker.go:1986] updateStatefulSet():Update StatefulSet(clickhouse/chi-analytics-cluster-0-0) - started
...
E0131 08:37:53.190819       1 creator.go:73] updateStatefulSet():StatefulSet.apps "chi-analytics-cluster-0-0" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden
E0131 08:37:53.190921       1 worker.go:2010] updateStatefulSet():Update StatefulSet(clickhouse/chi-analytics-cluster-0-0) - failed with error
...
Continue with recreate
I0131 08:37:53.259546       1 worker.go:2013] StatefulSet.Spec diff:
I0131 08:37:53.260260       1 worker.go:2014] AP item start -------------------------
modified spec items: 28
ap item path [0]:.Template.Spec.TerminationGracePeriodSeconds
ap item value[0]:'30'
ap item path [1]:.Template.Spec.Containers[0].LivenessProbe.TimeoutSeconds
ap item value[1]:'0'
ap item path [2]:.Template.Spec.Containers[0].ReadinessProbe.Handler.HTTPGet.Scheme
ap item value[2]:''
ap item path [3]:.Template.Spec.Containers[1].Ports[0].Protocol
ap item value[3]:''
ap item path [4]:.Template.Spec.Containers[1].TerminationMessagePath
ap item value[4]:''
ap item path [5]:.Template.Spec.RestartPolicy
ap item value[5]:''
ap item path [6]:.Template.Spec.Containers[0].LivenessProbe.Handler.HTTPGet.Scheme
ap item value[6]:''
ap item path [7]:.Template.Spec.Containers[0].ReadinessProbe.TimeoutSeconds
ap item value[7]:'0'
ap item path [8]:.Template.Spec.Containers[0].TerminationMessagePolicy
ap item value[8]:''
ap item path [9]:.Template.Spec.SchedulerName
ap item value[9]:''
ap item path [10]:.Template.Spec.SecurityContext
ap item value[10]:'nil'
ap item path [11]:.Template.Spec.Containers[0].Ports[2].Protocol
ap item value[11]:''
ap item path [12]:.Template.Spec.Containers[0].TerminationMessagePath
ap item value[12]:''
ap item path [13]:.Template.Spec.Containers[2].TerminationMessagePolicy
ap item value[13]:''
ap item path [14]:.Template.Spec.Containers[2].ImagePullPolicy
ap item value[14]:''
ap item path [15]:.Template.Spec.DNSPolicy
ap item value[15]:''
ap item path [16]:.Template.Spec.Containers[0].Ports[1].Protocol
ap item value[16]:''
ap item path [17]:.Template.Spec.Containers[0].Ports[0].Protocol
ap item value[17]:''
ap item path [18]:.Template.Spec.Containers[0].ReadinessProbe.FailureThreshold
ap item value[18]:'0'
ap item path [19]:.VolumeClaimTemplates[0].ObjectMeta.Annotations
ap item value[19]:'map[meta.helm.sh/release-name:clickhouse meta.helm.sh/release-namespace:clickhouse]'
ap item path [20]:.Template.Spec.Containers[0].ImagePullPolicy
ap item value[20]:''
ap item path [21]:.VolumeClaimTemplates[1].Status.Phase
ap item value[21]:''
ap item path [22]:.Template.Spec.Containers[0].ReadinessProbe.SuccessThreshold
ap item value[22]:'0'
ap item path [23]:.Template.Spec.Containers[2].TerminationMessagePath
ap item value[23]:''
ap item path [24]:.VolumeClaimTemplates[1].ObjectMeta.Annotations
ap item value[24]:'map[meta.helm.sh/release-name:clickhouse meta.helm.sh/release-namespace:clickhouse]'
ap item path [25]:.Template.Spec.Containers[0].LivenessProbe.SuccessThreshold
ap item value[25]:'0'
ap item path [26]:.Template.Spec.Containers[1].TerminationMessagePolicy
ap item value[26]:''
ap item path [27]:.VolumeClaimTemplates[0].Status.Phase
ap item value[27]:''
AP item end -------------------------
I0131 08:37:53.260943       1 deleter.go:126] deleteStatefulSet():clickhouse/chi-analytics-cluster-0-0
...
I0131 08:39:03.898964       1 worker.go:1895] createStatefulSet():Create StatefulSet clickhouse/chi-analytics-cluster-0-0 - started

Thanks,

[alex-zaitsev]

@tdminh , looks like your CHI is corrupted by a custom Helm chart. Or maybe you did not update the CRD. The log assumes your CHI does not match the spec.

[minhtd1981]

Hi @alex-zaitsev

Yes I created a helm chart to install CHI via Jenkins pipeline. Do you mean that I should not do that? The reason why we want to do is to keep the history of all changes to our K8S.

Thanks,

[washanhanzi]

What does this error mean?

W0131 08:36:52.285254       1 normalizer.go:926] unable to parse secret field address:

I also got this error from the operator.

[tdminh]

What does this error mean?

W0131 08:36:52.285254       1 normalizer.go:926] unable to parse secret field address:

I also got this error from the operator.

me too

[panboo0106]

What does this error mean?

W0131 08:36:52.285254       1 normalizer.go:926] unable to parse secret field address:

I also got this error from the operator.

how to solve it？

[panboo0106]

What does this error mean?

W0131 08:36:52.285254       1 normalizer.go:926] unable to parse secret field address:

I also got this error from the operator.

how to solve it？

I saw the history logs that there were some error about not connecting to the coredns service. I just restart pod, it works. I'm new here, I don't know why.So hope someone can explain. Thk