clickhouse-operator icon indicating copy to clipboard operation
clickhouse-operator copied to clipboard
verified publisher icon Altinity

Controller crashes when chi resources already exists and the CRD has been deleted

Open vtomasr5 opened this issue 3 years ago • 0 comments

The clickhouse-operator installed via Helm chart (https://github.com/slamdev/helm-charts/tree/master/charts/altinity-clickhouse-operator) crashes with the following output:

namespace: ""
name: ""
runtime:
	username: '***'
	password: '***'
port: 8123
template:
chi:
path: /etc/clickhouse-operator/templates.d
runtime:
templatefiles: {}
reconcile:
runtime:
threadsNumber: 10
statefulSet:
create:
onFailure: ignore
update:
timeout: 300
pollInterval: 5
onFailure: rollback
host:
host:
exclude: false
include: false
annotation:
include: []
exclude: []
label:
include: []
exclude: []
appendScope: "no"
runtime:
appendscope: false
statefulSet:
revisionHistoryLimit: 10
pod:
terminationGracePeriod: 30
logger:
logtostderr: "true"
alsologtostderr: "false"
v: "1"
stderrthreshold: ""
vmodule: ""
log_backtrace_at: ""
watchNamespaces: []
chCommonConfigsPath: ""
chHostConfigsPath: ""
chUsersConfigsPath: ""
chiTemplatesPath: ""
statefulSetUpdateTimeout: 0
statefulSetUpdatePollPeriod: 0
onStatefulSetCreateFailureAction: ""
onStatefulSetUpdateFailureAction: ""
chConfigUserDefaultProfile: ""
chConfigUserDefaultQuota: ""
chConfigUserDefaultNetworksIP: []
chConfigUserDefaultPassword: '***'
chConfigNetworksHostRegexpTemplate: ""
chScheme: ""
chUsername: '***'
chPassword: '***'
chCredentialsSecretNamespace: ""
chCredentialsSecretName: ""
chPort: 0
logtostderr: ""
alsologtostderr: ""
v: ""
stderrthreshold: ""
vmodule: ""
log_backtrace_at: ""
reconcileThreadsNumber: 0
reconcileWaitExclude: false
reconcileWaitInclude: false
includeIntoPropagationAnnotations: []
excludeFromPropagationAnnotations: []
includeIntoPropagationLabels: []
excludeFromPropagationLabels: []
appendScopeLabels: ""
terminationGracePeriod: 0
revisionHistoryLimit: 0
I0629 09:44:25.580485       1 clickhouse_operator.go:146] Run():Starting CHI controller
I0629 09:44:25.580535       1 controller.go:416] Starting ClickHouseInstallation controller
I0629 09:44:25.580553       1 controller.go:848] waitForCacheSync():Syncing caches for ClickHouseInstallation controller
I0629 09:44:25.617509       1 controller.go:517] ENQUEUE new ReconcileCHI cmd=add for clickhouse-analytics/clickhouse-analytics
I0629 09:44:25.680869       1 controller.go:853] waitForCacheSync():Caches are synced for ClickHouseInstallation controller
I0629 09:44:25.681193       1 labeler.go:80] OPERATOR_POD_NAMESPACE=clickhouse-operator OPERATOR_POD_NAME=clickhouse-operator-altinity-clickhouse-operator-7896fd8db49k2q
I0629 09:44:25.813341       1 controller.go:448] Run():ClickHouseInstallation controller: starting workers number: 11
I0629 09:44:25.813611       1 controller.go:450] Run():ClickHouseInstallation controller: starting worker 1 out of 11
I0629 09:44:25.813820       1 controller.go:450] Run():ClickHouseInstallation controller: starting worker 2 out of 11
I0629 09:44:25.814122       1 controller.go:450] Run():ClickHouseInstallation controller: starting worker 3 out of 11
I0629 09:44:25.814364       1 controller.go:450] Run():ClickHouseInstallation controller: starting worker 4 out of 11
I0629 09:44:25.814674       1 controller.go:450] Run():ClickHouseInstallation controller: starting worker 5 out of 11
I0629 09:44:25.814893       1 controller.go:450] Run():ClickHouseInstallation controller: starting worker 6 out of 11
I0629 09:44:25.815285       1 controller.go:450] Run():ClickHouseInstallation controller: starting worker 7 out of 11
I0629 09:44:25.815535       1 controller.go:450] Run():ClickHouseInstallation controller: starting worker 8 out of 11
I0629 09:44:25.815761       1 controller.go:450] Run():ClickHouseInstallation controller: starting worker 9 out of 11
I0629 09:44:25.815981       1 controller.go:450] Run():ClickHouseInstallation controller: starting worker 10 out of 11
I0629 09:44:25.816183       1 controller.go:450] Run():ClickHouseInstallation controller: starting worker 11 out of 11
I0629 09:44:25.816419       1 controller.go:460] Run():ClickHouseInstallation controller: workers started
I0629 15:51:33.112442       1 worker.go:1187] deleteCHI():clickhouse-analytics/clickhouse-analytics:CRD /clickhouseinstallations.clickhouse.altinity.com is not being deleted, operator will delete child resources
E0629 15:51:33.126895       1 runtime.go:76] Observed a panic: reflect: reflect.Value.Set using value obtained using unexported field
goroutine 151 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x15566e0, 0xc0007c4c30})
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x0})
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x75
panic({0x15566e0, 0xc0007c4c30})
        /usr/local/go/src/runtime/panic.go:1038 +0x215
reflect.flag.mustBeAssignableSlow(0x146fa64)
        /usr/local/go/src/reflect/value.go:259 +0xc5
reflect.flag.mustBeAssignable(...)
        /usr/local/go/src/reflect/value.go:249
reflect.Value.Set({0x1737580, 0xc00001dfc0, 0x203000}, {0x1737580, 0xc00001df80, 0xc0009d89b8})
        /usr/local/go/src/reflect/value.go:1912 +0x6a
github.com/imdario/mergo.deepMerge({0x1737580, 0xc00001dfc0, 0xc00069cdc8}, {0x1737580, 0xc00001df80, 0x1}, 0x1, 0x5, 0xc000224ae0)
        /clickhouse-operator/vendor/github.com/imdario/mergo/merge.go:99 +0xeb1
github.com/imdario/mergo.deepMerge({0x173eba0, 0xc00001dfc0, 0xcc00148a0c0}, {0x173eba0, 0xc00001df80, 0x98}, 0x1010000017d0ea0, 0x4, 0xc000224ae0)
        /clickhouse-operator/vendor/github.com/imdario/mergo/merge.go:93 +0xfd8
github.com/imdario/mergo.deepMerge({0x167b220, 0xc0001354c0, 0xcc00148a038}, {0x167b220, 0xc000135640, 0xc0009d9248}, 0x10100000041762d, 0x3, 0xc000224ae0)
        /clickhouse-operator/vendor/github.com/imdario/mergo/merge.go:147 +0x18a9
github.com/imdario/mergo.deepMerge({0x16a4be0, 0xc0001354b8, 0xcc001489e18}, {0x16a4be0, 0xc000135638, 0xc000224aa0}, 0x101000000000000, 0x2, 0xc000224ae0)
        /clickhouse-operator/vendor/github.com/imdario/mergo/merge.go:93 +0xfd8
github.com/imdario/mergo.deepMerge({0x173b240, 0xc000135498, 0xcc001488b80}, {0x173b240, 0xc000135618, 0x380}, 0x380, 0x1, 0xc000224ae0)
        /clickhouse-operator/vendor/github.com/imdario/mergo/merge.go:93 +0xfd8
github.com/imdario/mergo.deepMerge({0x16ed9a0, 0xc000135380, 0x16ed9a0}, {0x16ed9a0, 0xc000135500, 0x0}, 0x203000, 0x0, 0xc000224ae0)
        /clickhouse-operator/vendor/github.com/imdario/mergo/merge.go:93 +0xfd8
github.com/imdario/mergo.merge({0x160fbe0, 0xc000135380}, {0x16ed9a0, 0xc000135500}, {0xc0009d9ce0, 0x1, 0x0})
        /clickhouse-operator/vendor/github.com/imdario/mergo/merge.go:361 +0x2ab
github.com/imdario/mergo.Merge(...)
        /clickhouse-operator/vendor/github.com/imdario/mergo/merge.go:291
github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1.(*ChiTemplates).mergeVolumeClaimTemplates(0xc0004f3780, 0xc0006c5680)
        /clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_templates.go:160 +0x16f
github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1.(*ChiTemplates).MergeFrom(0x1833000, 0xc0006c5680, {0x17fe310, 0x8})
        /clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_templates.go:51 +0x85
github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1.(*ChiSpec).MergeFrom(0xc0001fef18, 0xc000740118, {0x17fe310, 0xc00067fc00})
        /clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_chi.go:527 +0x425
github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1.(*ClickHouseInstallation).MergeFrom(0xc0001fee00, 0xc000740000, {0x17fe310, 0xc0009da110})
        /clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_chi.go:476 +0x22a
github.com/altinity/clickhouse-operator/pkg/model.(*Normalizer).CreateTemplatedCHI(0xc0004895a0, 0x0)
        /clickhouse-operator/pkg/model/normalizer.go:131 +0x11e5
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).deleteCHIProtocol(0xc0000df400, {0x1a25f78, 0xc000882680}, 0xc000740000)
        /clickhouse-operator/pkg/controller/chi/worker.go:1259 +0x5db
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).deleteCHI(0xc0000df400, {0x1a25f78, 0xc000882680}, 0xc000740000, 0xc000740000)
        /clickhouse-operator/pkg/controller/chi/worker.go:1214 +0x1371
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).updateCHI(0xc0000df400, {0x1a25f78, 0xc000882680}, 0x0, 0xc000740000)
        /clickhouse-operator/pkg/controller/chi/worker.go:278 +0x76b
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).processReconcileCHI(0x100000003, {0x1a25f78, 0xc000882680}, 0x0)
        /clickhouse-operator/pkg/controller/chi/worker.go:124 +0x48
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).processItem(0xc0000df400, {0x1a25f78, 0xc000882680}, {0x16107e0, 0xc000651500})
        /clickhouse-operator/pkg/controller/chi/worker.go:191 +0x51d
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).run(0xc0000df400)
        /clickhouse-operator/pkg/controller/chi/worker.go:105 +0x434
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7fd98cf85f98)
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0, {0x19f9e40, 0xc000674a20}, 0x1, 0xc0000900c0)
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0x0, 0x0)
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x0, 0x0, 0x0)
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x25
created by github.com/altinity/clickhouse-operator/pkg/controller/chi.(*Controller).Run
        /clickhouse-operator/pkg/controller/chi/controller.go:456 +0x7c5
panic: reflect: reflect.Value.Set using value obtained using unexported field [recovered]
        panic: reflect: reflect.Value.Set using value obtained using unexported field

goroutine 151 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x0})
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0xd8
panic({0x15566e0, 0xc0007c4c30})
        /usr/local/go/src/runtime/panic.go:1038 +0x215
reflect.flag.mustBeAssignableSlow(0x146fa64)
        /usr/local/go/src/reflect/value.go:259 +0xc5
reflect.flag.mustBeAssignable(...)
        /usr/local/go/src/reflect/value.go:249
reflect.Value.Set({0x1737580, 0xc00001dfc0, 0x203000}, {0x1737580, 0xc00001df80, 0xc0009d89b8})
        /usr/local/go/src/reflect/value.go:1912 +0x6a
github.com/imdario/mergo.deepMerge({0x1737580, 0xc00001dfc0, 0xc00069cdc8}, {0x1737580, 0xc00001df80, 0x1}, 0x1, 0x5, 0xc000224ae0)
        /clickhouse-operator/vendor/github.com/imdario/mergo/merge.go:99 +0xeb1
github.com/imdario/mergo.deepMerge({0x173eba0, 0xc00001dfc0, 0xcc00148a0c0}, {0x173eba0, 0xc00001df80, 0x98}, 0x1010000017d0ea0, 0x4, 0xc000224ae0)
        /clickhouse-operator/vendor/github.com/imdario/mergo/merge.go:93 +0xfd8
github.com/imdario/mergo.deepMerge({0x167b220, 0xc0001354c0, 0xcc00148a038}, {0x167b220, 0xc000135640, 0xc0009d9248}, 0x10100000041762d, 0x3, 0xc000224ae0)
        /clickhouse-operator/vendor/github.com/imdario/mergo/merge.go:147 +0x18a9
github.com/imdario/mergo.deepMerge({0x16a4be0, 0xc0001354b8, 0xcc001489e18}, {0x16a4be0, 0xc000135638, 0xc000224aa0}, 0x101000000000000, 0x2, 0xc000224ae0)
        /clickhouse-operator/vendor/github.com/imdario/mergo/merge.go:93 +0xfd8
github.com/imdario/mergo.deepMerge({0x173b240, 0xc000135498, 0xcc001488b80}, {0x173b240, 0xc000135618, 0x380}, 0x380, 0x1, 0xc000224ae0)
        /clickhouse-operator/vendor/github.com/imdario/mergo/merge.go:93 +0xfd8
github.com/imdario/mergo.deepMerge({0x16ed9a0, 0xc000135380, 0x16ed9a0}, {0x16ed9a0, 0xc000135500, 0x0}, 0x203000, 0x0, 0xc000224ae0)
        /clickhouse-operator/vendor/github.com/imdario/mergo/merge.go:93 +0xfd8
github.com/imdario/mergo.merge({0x160fbe0, 0xc000135380}, {0x16ed9a0, 0xc000135500}, {0xc0009d9ce0, 0x1, 0x0})
        /clickhouse-operator/vendor/github.com/imdario/mergo/merge.go:361 +0x2ab
github.com/imdario/mergo.Merge(...)
        /clickhouse-operator/vendor/github.com/imdario/mergo/merge.go:291
github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1.(*ChiTemplates).mergeVolumeClaimTemplates(0xc0004f3780, 0xc0006c5680)
        /clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_templates.go:160 +0x16f
github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1.(*ChiTemplates).MergeFrom(0x1833000, 0xc0006c5680, {0x17fe310, 0x8})
        /clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_templates.go:51 +0x85
github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1.(*ChiSpec).MergeFrom(0xc0001fef18, 0xc000740118, {0x17fe310, 0xc00067fc00})
        /clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_chi.go:527 +0x425
github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1.(*ClickHouseInstallation).MergeFrom(0xc0001fee00, 0xc000740000, {0x17fe310, 0xc0009da110})
        /clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_chi.go:476 +0x22a
github.com/altinity/clickhouse-operator/pkg/model.(*Normalizer).CreateTemplatedCHI(0xc0004895a0, 0x0)
        /clickhouse-operator/pkg/model/normalizer.go:131 +0x11e5
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).deleteCHIProtocol(0xc0000df400, {0x1a25f78, 0xc000882680}, 0xc000740000)
        /clickhouse-operator/pkg/controller/chi/worker.go:1259 +0x5db
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).deleteCHI(0xc0000df400, {0x1a25f78, 0xc000882680}, 0xc000740000, 0xc000740000)
        /clickhouse-operator/pkg/controller/chi/worker.go:1214 +0x1371
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).updateCHI(0xc0000df400, {0x1a25f78, 0xc000882680}, 0x0, 0xc000740000)
        /clickhouse-operator/pkg/controller/chi/worker.go:278 +0x76b
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).processReconcileCHI(0x100000003, {0x1a25f78, 0xc000882680}, 0x0)
        /clickhouse-operator/pkg/controller/chi/worker.go:124 +0x48
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).processItem(0xc0000df400, {0x1a25f78, 0xc000882680}, {0x16107e0, 0xc000651500})
        /clickhouse-operator/pkg/controller/chi/worker.go:191 +0x51d
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).run(0xc0000df400)
        /clickhouse-operator/pkg/controller/chi/worker.go:105 +0x434
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7fd98cf85f98)
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0, {0x19f9e40, 0xc000674a20}, 0x1, 0xc0000900c0)
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0x0, 0x0)
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x0, 0x0, 0x0)
        /clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x25
created by github.com/altinity/clickhouse-operator/pkg/controller/chi.(*Controller).Run
        /clickhouse-operator/pkg/controller/chi/controller.go:456 +0x7c5

This happens when the CRD does not exists anymore but there are chi resources still present. The controller doesn't handle that case properly and therefore the crash above.

Thanks

vtomasr5 avatar Jun 29 '22 16:06 vtomasr5