clickhouse-operator
clickhouse-operator copied to clipboard
Published
20 hours ago •
Altinity
Altinity
Failure to resize PVC after CHI recreation
I delete CHI and recreate it; after I tried to resize PVC but it's not happens; in operator log error
Operation cannot be fulfilled on persistentvolumeclaims "data-volume-template-chi-clickhouse-default-0-0-0": the object has been modified; please apply your changes to the latest version and try again With clean setup of (no PVC exists at time of CHI creation) PVC resized as intendent.
Operator version: 0.21.3
Steps to reproduce:
- create CHI
- delete CHI (to check data do not deleted)
- recreate CHI
- resize PVC
Chi:
spec:
configuration:
clusters:
- layout:
replicasCount: 1
shardsCount: 1
name: default
schemaPolicy:
replica: All
shard: All
defaults:
storageManagement:
provisioner: Operator
reclaimPolicy: Retain
templates:
dataVolumeClaimTemplate: data-volume-template
serviceTemplate: default-service-template
templates:
volumeClaimTemplates:
- name: data-volume-template
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 15G
useTemplates:
- name: secure-svc
Logs of failed state:
I1027 16:16:32.811467 1 worker.go:521] ActionPlan start---------------------------------------------:
AP item start -------------------------
modified spec items: 4
ap item path [0]:'.Templates.VolumeClaimTemplates[0].Spec.Resources.Requests["storage"].s'
ap item value[0]:'"15G"'
ap item path [1]:'.TaskID'
ap item value[1]:'"e321266b-7707-47ba-9a19-eac2b25e103e"'
ap item path [2]:'.Templates.VolumeClaimTemplates[0].Spec.Resources.Requests["storage"].i.value'
ap item value[2]:'15'
ap item path [3]:'.Templates.VolumeClaimTemplates[0].Spec.Resources.Requests["storage"].i.scale'
ap item value[3]:'9'
AP item end -------------------------
ActionPlan end---------------------------------------------
I1027 16:16:32.811573 1 worker-chi-reconciler.go:69] reconcileCHI():clickhouse/clickhouse/e321266b-7707-47ba-9a19-eac2b25e103e:ActionPlan has actions - continue reconcile
I1027 16:16:32.856488 1 worker.go:599] markReconcileStart():clickhouse/clickhouse/e321266b-7707-47ba-9a19-eac2b25e103e:reconcile started, task id: e321266b-7707-47ba-9a19-eac2b25e103e
I1027 16:16:32.977231 1 worker.go:744] FOUND host: ns:clickhouse|chi:clickhouse|clu:default|sha:0|rep:0|host:0-0
I1027 16:16:32.977308 1 worker.go:772] RemoteServersGeneratorOptions: exclude hosts: [], attributes: status: , add: true, remove: false, modify: false, found: false, excluded host(s):
I1027 16:16:32.987385 1 worker.go:1091] updateConfigMap():clickhouse/clickhouse/e321266b-7707-47ba-9a19-eac2b25e103e:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I1027 16:16:33.395964 1 worker.go:1091] updateConfigMap():clickhouse/clickhouse/e321266b-7707-47ba-9a19-eac2b25e103e:Update ConfigMap clickhouse/chi-clickhouse-common-usersd
I1027 16:16:33.827158 1 creator.go:113] CreateServiceCluster():clickhouse/clickhouse/e321266b-7707-47ba-9a19-eac2b25e103e:clickhouse/cluster-clickhouse-default
I1027 16:16:33.840218 1 worker-chi-reconciler.go:566] PDB updated clickhouse/clickhouse-default
I1027 16:16:33.840388 1 creator.go:136] CreateServiceShard():clickhouse/clickhouse/e321266b-7707-47ba-9a19-eac2b25e103e:clickhouse/shard-clickhouse-default-0
I1027 16:16:33.840498 1 worker-chi-reconciler.go:498] reconcileHost():Reconcile Host 0-0 started
I1027 16:16:34.226825 1 creator.go:593] getPodTemplate():clickhouse/clickhouse/e321266b-7707-47ba-9a19-eac2b25e103e:statefulSet chi-clickhouse-default-0-0 use default generated template
I1027 16:16:34.230340 1 worker.go:1279] getStatefulSetStatus():clickhouse/chi-clickhouse-default-0-0:cur and new StatefulSets ARE EQUAL based on labels. No StatefulSet reconcile is required for: clickhouse/chi-clickhouse-default-0-0
I1027 16:16:34.230720 1 worker.go:134] shouldForceRestartHost():Force restart is not required. Host: 0-0
I1027 16:16:34.230942 1 worker.go:978] shouldExcludeHost():The same host would not be updated host 0 shard 0 cluster default
I1027 16:16:34.398517 1 worker.go:1091] updateConfigMap():clickhouse/clickhouse/e321266b-7707-47ba-9a19-eac2b25e103e:Update ConfigMap clickhouse/chi-clickhouse-deploy-confd-default-0-0
E1027 16:16:35.399970 1 creator.go:175] updatePersistentVolumeClaim():clickhouse/data-volume-template-chi-clickhouse-default-0-0-0:unable to Update PVC err: Operation cannot be fulfilled on persistentvolumeclaims "data-volume-template-chi-clickhouse-default-0-0-0": the object has been modified; please apply your changes to the latest version and try again
E1027 16:16:35.400026 1 worker-chi-reconciler.go:802] func1():ERROR unable to reconcile PVC(clickhouse/data-volume-template-chi-clickhouse-default-0-0-0) err: Operation cannot be fulfilled on persistentvolumeclaims "data-volume-template-chi-clickhouse-default-0-0-0": the object has been modified; please apply your changes to the latest version and try again
I1027 16:16:35.400105 1 cluster.go:84] Run query on: chi-clickhouse-default-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-default-0-0.clickhouse.svc.cluster.local]
I1027 16:16:35.416949 1 worker-chi-reconciler.go:292] reconcileHostStatefulSet():Reconcile host 0-0. ClickHouse version: 23.9.2.56
I1027 16:16:35.417116 1 worker.go:134] shouldForceRestartHost():Force restart is not required. Host: 0-0
I1027 16:16:35.417135 1 worker-chi-reconciler.go:304] reconcileHostStatefulSet():Reconcile host 0-0. Reconcile StatefulSet
I1027 16:16:35.417278 1 creator.go:593] getPodTemplate():clickhouse/clickhouse/e321266b-7707-47ba-9a19-eac2b25e103e:statefulSet chi-clickhouse-default-0-0 use default generated template
I1027 16:16:35.418046 1 worker.go:1279] getStatefulSetStatus():clickhouse/chi-clickhouse-default-0-0:cur and new StatefulSets ARE EQUAL based on labels. No StatefulSet reconcile is required for: clickhouse/chi-clickhouse-default-0-0
E1027 16:16:35.996637 1 creator.go:175] updatePersistentVolumeClaim():clickhouse/data-volume-template-chi-clickhouse-default-0-0-0:unable to Update PVC err: Operation cannot be fulfilled on persistentvolumeclaims "data-volume-template-chi-clickhouse-default-0-0-0": the object has been modified; please apply your changes to the latest version and try again
E1027 16:16:35.996688 1 worker-chi-reconciler.go:802] func1():ERROR unable to reconcile PVC(clickhouse/data-volume-template-chi-clickhouse-default-0-0-0) err: Operation cannot be fulfilled on persistentvolumeclaims "data-volume-template-chi-clickhouse-default-0-0-0": the object has been modified; please apply your changes to the latest version and try again
Clean install logs:
I1027 16:37:54.553220 1 controller.go:566] ENQUEUE new ReconcileCHI cmd=update for clickhouse/clickhouse
I1027 16:37:54.553329 1 worker.go:381] worker.go:381:updateCHI():start:clickhouse/clickhouse
I1027 16:37:54.553363 1 worker.go:423] Operator IPs. Previous: 10.244.0.132 Cur: 10.244.0.132
I1027 16:37:54.553370 1 worker.go:432] Operator IPs are the same. It is restart on the same IP
I1027 16:37:54.553378 1 worker.go:440] Operator is not just started. May not be clean restart
I1027 16:37:54.555763 1 worker-chi-reconciler.go:51] worker-chi-reconciler.go:51:reconcileCHI():start:clickhouse/clickhouse
I1027 16:37:54.555804 1 worker-chi-reconciler.go:55] reconcileCHI():clickhouse/clickhouse:has ancestor, use it as a base for reconcile
I1027 16:37:54.555834 1 normalizer.go:137] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Found auto-templates num: 2
I1027 16:37:54.555848 1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Adding auto-template to list of possible templates: clickhouse/no-logs
I1027 16:37:54.555860 1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Adding auto-template to list of possible templates: clickhouse/secure-ssl
I1027 16:37:54.555901 1 normalizer.go:149] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Found manual-templates num: 1
I1027 16:37:54.555921 1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Apply template: clickhouse/no-logs. Selector: map[] matches labels: map[]
I1027 16:37:54.555951 1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Apply template: clickhouse/secure-ssl. Selector: map[] matches labels: map[]
I1027 16:37:54.555974 1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Apply template: /secure-svc. Selector: map[] matches labels: map[]
I1027 16:37:54.555997 1 normalizer.go:202] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Used templates count: 3
I1027 16:37:54.567241 1 worker.go:291] clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:IPs of the CHI clickhouse/clickhouse: [10.244.1.54]
I1027 16:37:54.567300 1 normalizer.go:137] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Found auto-templates num: 2
I1027 16:37:54.567317 1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Adding auto-template to list of possible templates: clickhouse/no-logs
I1027 16:37:54.567332 1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Adding auto-template to list of possible templates: clickhouse/secure-ssl
I1027 16:37:54.567359 1 normalizer.go:149] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Found manual-templates num: 1
I1027 16:37:54.567381 1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Apply template: clickhouse/no-logs. Selector: map[] matches labels: map[]
I1027 16:37:54.567409 1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Apply template: clickhouse/secure-ssl. Selector: map[] matches labels: map[]
I1027 16:37:54.567441 1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Apply template: /secure-svc. Selector: map[] matches labels: map[]
I1027 16:37:54.567466 1 normalizer.go:202] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Used templates count: 3
I1027 16:37:54.568557 1 normalizer.go:137] applyCHITemplates():clickhouse/clickhouse:Found auto-templates num: 2
I1027 16:37:54.568577 1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse:Adding auto-template to list of possible templates: clickhouse/no-logs
I1027 16:37:54.568596 1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse:Adding auto-template to list of possible templates: clickhouse/secure-ssl
I1027 16:37:54.568609 1 normalizer.go:149] applyCHITemplates():clickhouse/clickhouse:Found manual-templates num: 1
I1027 16:37:54.568624 1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: clickhouse/no-logs. Selector: map[] matches labels: map[]
I1027 16:37:54.568648 1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: clickhouse/secure-ssl. Selector: map[] matches labels: map[]
I1027 16:37:54.568678 1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: /secure-svc. Selector: map[] matches labels: map[]
I1027 16:37:54.568700 1 normalizer.go:202] applyCHITemplates():clickhouse/clickhouse:Used templates count: 3
I1027 16:37:54.576822 1 worker.go:291] clickhouse/clickhouse/e8060938-721d-4df0-9685-1aeb61f7825e:IPs of the CHI clickhouse/clickhouse: [10.244.1.54]
I1027 16:37:54.576933 1 normalizer.go:137] applyCHITemplates():clickhouse/clickhouse:Found auto-templates num: 2
I1027 16:37:54.577014 1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse:Adding auto-template to list of possible templates: clickhouse/no-logs
I1027 16:37:54.577202 1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse:Adding auto-template to list of possible templates: clickhouse/secure-ssl
I1027 16:37:54.577231 1 normalizer.go:149] applyCHITemplates():clickhouse/clickhouse:Found manual-templates num: 1
I1027 16:37:54.577283 1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: clickhouse/no-logs. Selector: map[] matches labels: map[]
I1027 16:37:54.577447 1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: clickhouse/secure-ssl. Selector: map[] matches labels: map[]
I1027 16:37:54.577590 1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: /secure-svc. Selector: map[] matches labels: map[]
I1027 16:37:54.577721 1 normalizer.go:202] applyCHITemplates():clickhouse/clickhouse:Used templates count: 3
I1027 16:37:54.583181 1 worker.go:521] ActionPlan start---------------------------------------------:
AP item start -------------------------
modified spec items: 3
ap item path [0]:'.TaskID'
ap item value[0]:'"c14edf04-9813-4f56-8531-b93831dbbb04"'
ap item path [1]:'.Templates.VolumeClaimTemplates[0].Spec.Resources.Requests["storage"].i.value'
ap item value[1]:'15'
ap item path [2]:'.Templates.VolumeClaimTemplates[0].Spec.Resources.Requests["storage"].s'
ap item value[2]:'"15G"'
AP item end -------------------------
ActionPlan end---------------------------------------------
I1027 16:37:54.583332 1 worker-chi-reconciler.go:69] reconcileCHI():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:ActionPlan has actions - continue reconcile
I1027 16:37:54.617555 1 worker.go:599] markReconcileStart():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:reconcile started, task id: c14edf04-9813-4f56-8531-b93831dbbb04
I1027 16:37:54.709974 1 worker.go:744] FOUND host: ns:clickhouse|chi:clickhouse|clu:default|sha:0|rep:0|host:0-0
I1027 16:37:54.710036 1 worker.go:772] RemoteServersGeneratorOptions: exclude hosts: [], attributes: status: , add: true, remove: false, modify: false, found: false, excluded host(s):
I1027 16:37:54.719919 1 worker.go:1091] updateConfigMap():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I1027 16:37:55.172462 1 worker.go:1091] updateConfigMap():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Update ConfigMap clickhouse/chi-clickhouse-common-usersd
I1027 16:37:55.600303 1 creator.go:113] CreateServiceCluster():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:clickhouse/cluster-clickhouse-default
I1027 16:37:55.616927 1 worker-chi-reconciler.go:566] PDB updated clickhouse/clickhouse-default
I1027 16:37:55.617044 1 creator.go:136] CreateServiceShard():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:clickhouse/shard-clickhouse-default-0
I1027 16:37:55.617104 1 worker-chi-reconciler.go:498] reconcileHost():Reconcile Host 0-0 started
I1027 16:37:56.005130 1 creator.go:593] getPodTemplate():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:statefulSet chi-clickhouse-default-0-0 use default generated template
I1027 16:37:56.006800 1 worker.go:1279] getStatefulSetStatus():clickhouse/chi-clickhouse-default-0-0:cur and new StatefulSets ARE EQUAL based on labels. No StatefulSet reconcile is required for: clickhouse/chi-clickhouse-default-0-0
I1027 16:37:56.007308 1 worker.go:134] shouldForceRestartHost():Force restart is not required. Host: 0-0
I1027 16:37:56.007337 1 worker.go:978] shouldExcludeHost():The same host would not be updated host 0 shard 0 cluster default
I1027 16:37:56.169120 1 worker.go:1091] updateConfigMap():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Update ConfigMap clickhouse/chi-clickhouse-deploy-confd-default-0-0
I1027 16:37:57.184749 1 cluster.go:84] Run query on: chi-clickhouse-default-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-default-0-0.clickhouse.svc.cluster.local]
I1027 16:37:57.200861 1 worker-chi-reconciler.go:292] reconcileHostStatefulSet():Reconcile host 0-0. ClickHouse version: 23.9.2.56
I1027 16:37:57.201167 1 worker.go:134] shouldForceRestartHost():Force restart is not required. Host: 0-0
I1027 16:37:57.201222 1 worker-chi-reconciler.go:304] reconcileHostStatefulSet():Reconcile host 0-0. Reconcile StatefulSet
I1027 16:37:57.201375 1 creator.go:593] getPodTemplate():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:statefulSet chi-clickhouse-default-0-0 use default generated template
I1027 16:37:57.202535 1 worker.go:1279] getStatefulSetStatus():clickhouse/chi-clickhouse-default-0-0:cur and new StatefulSets ARE EQUAL based on labels. No StatefulSet reconcile is required for: clickhouse/chi-clickhouse-default-0-0
I1027 16:37:57.772604 1 creator.go:160] CreateServiceHost():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:clickhouse/chi-clickhouse-default-0-0 for Set chi-clickhouse-default-0-0
I1027 16:37:57.968558 1 worker.go:1203] updateService():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Update Service clickhouse/chi-clickhouse-default-0-0
I1027 16:37:58.397772 1 worker.go:802] migrateTables():No need to add tables on host 0 to shard 0 in cluster default
I1027 16:37:58.397868 1 cluster.go:84] Run query on: chi-clickhouse-default-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-default-0-0.clickhouse.svc.cluster.local]
I1027 16:37:58.405791 1 worker.go:887] includeHost():Include into cluster host 0 shard 0 cluster default
I1027 16:37:58.405821 1 worker.go:772] RemoteServersGeneratorOptions: exclude hosts: [], attributes: status: , add: true, remove: false, modify: false, found: false, excluded host(s):
I1027 16:37:58.570730 1 worker.go:1091] updateConfigMap():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I1027 16:37:59.168727 1 worker-chi-reconciler.go:537] reconcileHost():Reconcile Host 0-0 completed. ClickHouse version running: 23.9.2.56
I1027 16:37:59.603804 1 worker-chi-reconciler.go:553] reconcileHost():ProgressHostsCompleted: 1 of 1
I1027 16:38:00.000539 1 creator.go:58] CreateServiceCHI():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:clickhouse/clickhouse-clickhouse
I1027 16:38:00.168887 1 worker.go:1203] updateService():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Update Service clickhouse/clickhouse-clickhouse
I1027 16:38:00.771738 1 worker.go:1091] updateConfigMap():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I1027 16:38:01.203713 1 worker-chi-reconciler.go:103] reconcileCHI():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:remove items scheduled for deletion
I1027 16:38:01.605728 1 worker-deleter.go:37] clean():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Failed to reconcile objects:
I1027 16:38:01.605769 1 worker-deleter.go:38] clean():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Reconciled objects:
Service: clickhouse/chi-clickhouse-default-0-0
Service: clickhouse/clickhouse-clickhouse
ConfigMap: clickhouse/chi-clickhouse-common-usersd
ConfigMap: clickhouse/chi-clickhouse-deploy-confd-default-0-0
ConfigMap: clickhouse/chi-clickhouse-common-configd
PDB: clickhouse/clickhouse-default
PVC: clickhouse/data-volume-template-chi-clickhouse-default-0-0-0
StatefulSet: clickhouse/chi-clickhouse-default-0-0
I1027 16:38:02.373859 1 worker-deleter.go:41] clean():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Existing objects:
StatefulSet: clickhouse/chi-clickhouse-default-0-0
ConfigMap: clickhouse/chi-clickhouse-common-configd
ConfigMap: clickhouse/chi-clickhouse-common-usersd
ConfigMap: clickhouse/chi-clickhouse-deploy-confd-default-0-0
Service: clickhouse/chi-clickhouse-default-0-0
Service: clickhouse/clickhouse-clickhouse
PVC: clickhouse/data-volume-template-chi-clickhouse-default-0-0-0
PDB: clickhouse/clickhouse-default
I1027 16:38:02.373935 1 worker-deleter.go:43] clean():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Non-reconciled objects:
I1027 16:38:02.373955 1 worker-deleter.go:53] clean():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:remove items scheduled for deletion
I1027 16:38:02.799541 1 worker-deleter.go:64] worker-deleter.go:64:dropReplicas():start:clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:drop replicas based on AP
I1027 16:38:02.799584 1 worker-deleter.go:81] worker-deleter.go:81:dropReplicas():end:clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:processed replicas: 0
I1027 16:38:02.799632 1 worker.go:575] includeStopped():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:add CHI to monitoring
I1027 16:38:03.204465 1 controller.go:610] OK update watch (clickhouse/clickhouse): {"namespace":"clickhouse","name":"clickhouse","clusters":[{"name":"default","hosts":[{"name":"0-0","hostname":"chi-clickhouse-default-0-0.clickhouse.svc.cluster.local","tcpPort":9000,"tlsPort":9440,"httpPort":8123,"httpsPort":8443}]}]}
I1027 16:38:03.213330 1 worker.go:542] clickhouse/clickhouse:all IP addresses are in place
I1027 16:38:03.225662 1 normalizer.go:137] applyCHITemplates():clickhouse/clickhouse:Found auto-templates num: 2
I1027 16:38:03.225700 1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse:Adding auto-template to list of possible templates: clickhouse/no-logs
I1027 16:38:03.225713 1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse:Adding auto-template to list of possible templates: clickhouse/secure-ssl
I1027 16:38:03.225726 1 normalizer.go:149] applyCHITemplates():clickhouse/clickhouse:Found manual-templates num: 1
I1027 16:38:03.226347 1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: clickhouse/no-logs. Selector: map[] matches labels: map[]
I1027 16:38:03.226492 1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: clickhouse/secure-ssl. Selector: map[] matches labels: map[]
I1027 16:38:03.226595 1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: /secure-svc. Selector: map[] matches labels: map[]
I1027 16:38:03.226721 1 normalizer.go:202] applyCHITemplates():clickhouse/clickhouse:Used templates count: 3
I1027 16:38:03.367766 1 worker.go:613] clickhouse/clickhouse/833af56c-e246-4967-a07e-6ee2090469b4:IPs of the CHI-2 [10.244.1.54]
I1027 16:38:03.375655 1 normalizer.go:137] applyCHITemplates():clickhouse/clickhouse:Found auto-templates num: 2
I1027 16:38:03.375690 1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse:Adding auto-template to list of possible templates: clickhouse/no-logs
I1027 16:38:03.375701 1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse:Adding auto-template to list of possible templates: clickhouse/secure-ssl
I1027 16:38:03.375713 1 normalizer.go:149] applyCHITemplates():clickhouse/clickhouse:Found manual-templates num: 1
I1027 16:38:03.375904 1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: clickhouse/no-logs. Selector: map[] matches labels: map[]
I1027 16:38:03.375940 1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: clickhouse/secure-ssl. Selector: map[] matches labels: map[]
I1027 16:38:03.375964 1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: /secure-svc. Selector: map[] matches labels: map[]
I1027 16:38:03.375993 1 normalizer.go:202] applyCHITemplates():clickhouse/clickhouse:Used templates count: 3
I1027 16:38:03.377358 1 worker.go:617] clickhouse/clickhouse/33c5e95d-4f82-4f11-be40-7c0387663829:Update users IPS-2
I1027 16:38:03.597744 1 worker.go:638] finalizeReconcileAndMarkCompleted():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:reconcile completed successfully, task id: c14edf04-9813-4f56-8531-b93831dbbb04
I1027 16:38:03.993153 1 worker-chi-reconciler.go:111] worker-chi-reconciler.go:52:reconcileCHI():end:clickhouse/clickhouse
I1027 16:38:03.993190 1 worker.go:416] worker.go:382:updateCHI():end:clickhouse/clickhouse
Just to add, I have used a recent version of the operator from master. The change of the requested size is read without SI suffix. So, the change from 300Gi -> 350Gi results in some strange storage request size of 350.
