shell-operator icon indicating copy to clipboard operation
shell-operator copied to clipboard
verified publisher icon flant

Panic causing CrashLoopBackOff.

Open GrahamDumpleton opened this issue 3 years ago • 2 comments

Expected behavior (what you expected to happen):

No crash. :-)

Actual behavior (what actually happened):

Am still trying to narrow down the cause, but relates to using KUBERNETES_PATCH_PATH to have the Shell-operator apply changes back to the cluster. Something in a resource that is being passed back is causing the Shell-operator to crash. The error logs are:

time="2022-01-27T01:42:40Z" level=error msg="E0127 01:42:40.326099       7 runtime.go:78] Observed a panic: &errors.errorString{s:\"cannot deep copy int\"} (cannot deep copy int)\ngoroutine 9 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic(0x17c1a40, 0xc00070fe00)\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0xa6\nk8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x86\npanic(0x17c1a40, 0xc00070fe00)\n\t/usr/local/go/src/runtime/panic.go:965 +0x1b9\nk8s.io/apimachinery/pkg/runtime.DeepCopyJSONValue(0x177ac20, 0x267bde8, 0xc0008ae940, 0xc00054d440)\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/converter.go:466 +0x550\nk8s.io/apimachinery/pkg/runtime.DeepCopyJSONValue(0x17d7b40, 0xc0004ff230, 0xc0006335d0, 0x8)\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/converter.go:450 +0x177\nk8s.io/apimachinery/pkg/runtime.DeepCopyJSONValue(0x17d7b40, 0xc0004ff1a0, 0xc8, 0x1bc50a0)\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/converter.go:450 +0x177\nk8s.io/apimachinery/pkg/runtime.DeepCopyJSON(...)\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/converter.go:435\nk8s.io/apimachinery/pkg/apis/meta/v1/unstructured.(*Unstructured).DeepCopy(0xc000010290, 0x5)\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/apis/meta/v1/unstructured/unstructured.go:146 +0x75\ngithub.com/flant/shell-operator/pkg/kube/object_patch.(*ObjectPatcher).executeCreateOperation.func2(0xa93c8f, 0xc0005d5540)\n\t/app/pkg/kube/object_patch/patch.go:129 +0x3fa\nk8s.io/client-go/util/retry.OnError.func1(0x0, 0x0, 0x0)\n\t/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:51 +0x3c\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0xc0008aee18, 0x100, 0x0, 0x0)\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:211 +0x69\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0, 0xc0008aee18, 0x0, 0x0)\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:399 +0x55\nk8s.io/client-go/util/retry.OnError(0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0, 0x1a881d0, 0xc0008af050, 0x0, 0x0)\n\t/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:50 +0xa6\nk8s.io/client-go/util/retry.RetryOnConflict(...)\n\t/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:104\ngithub.com/flant/shell-operator/pkg/kube/object_patch.(*ObjectPatcher).executeCreateOperation(0xc0002c6978, 0xc000720840, 0x1bc4601, 0xc00070f030)\n\t/app/pkg/kube/object_patch/patch.go:118 +0x745\ngithub.com/flant/shell-operator/pkg/kube/object_patch.(*ObjectPatcher).ExecuteOperation(0xc0002c6978, 0x1bc38a0, 0xc000720840, 0x16, 0xc0008af170)\n\t/app/pkg/kube/object_patch/patch.go:65 +0x130\ngithub.com/flant/shell-operator/pkg/kube/object_patch.(*ObjectPatcher).ExecuteOperations(0xc0002c6978, 0xc0007d9100, 0x6, 0x8, 0x0, 0x0)\n\t/app/pkg/kube/object_patch/patch.go:49 +0x236\ngithub.com/flant/shell-operator/pkg/shell-operator.(*ShellOperator).HandleRunHook(0xc0002bd520, 0x1c12f68, 0xc000098280, 0xc000457cc0, 0xc0002cf4e4, 0x11, 0xc0006f0b10, 0xf, 0x0, 0x0, ...)\n\t/app/pkg/shell-operator/operator.go:761 +0x485\ngithub.com/flant/shell-operator/pkg/shell-operator.(*ShellOperator).TaskHandleHookRun(0xc0002bd520, 0x1c12f68, 0xc000098280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/app/pkg/shell-operator/operator.go:708 +0xbe8\ngithub.com/flant/shell-operator/pkg/shell-operator.(*ShellOperator).TaskHandler(0xc0002bd520, 0x1c12f68, 0xc000098280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/app/pkg/shell-operator/operator.go:538 +0x1f0\ngithub.com/flant/shell-operator/pkg/task/queue.(*TaskQueue).Start.func1(0xc000698420)\n\t/app/pkg/task/queue/task_queue.go:342 +0x3fc\ncreated by github.com/flant/shell-operator/pkg/task/queue.(*TaskQueue).Start\n\t/app/pkg/task/queue/task_queue.go:320 +0x4f\n" source=klog
panic: cannot deep copy int [recovered]
	panic: cannot deep copy int

goroutine 9 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x109
panic(0x17c1a40, 0xc00070fe00)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
k8s.io/apimachinery/pkg/runtime.DeepCopyJSONValue(0x177ac20, 0x267bde8, 0xc0008ae940, 0xc00054d440)
	/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/converter.go:466 +0x550
k8s.io/apimachinery/pkg/runtime.DeepCopyJSONValue(0x17d7b40, 0xc0004ff230, 0xc0006335d0, 0x8)
	/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/converter.go:450 +0x177
k8s.io/apimachinery/pkg/runtime.DeepCopyJSONValue(0x17d7b40, 0xc0004ff1a0, 0xc8, 0x1bc50a0)
	/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/converter.go:450 +0x177
k8s.io/apimachinery/pkg/runtime.DeepCopyJSON(...)
	/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/converter.go:435
k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.(*Unstructured).DeepCopy(0xc000010290, 0x5)
	/go/pkg/mod/k8s.io/[email protected]/pkg/apis/meta/v1/unstructured/unstructured.go:146 +0x75
github.com/flant/shell-operator/pkg/kube/object_patch.(*ObjectPatcher).executeCreateOperation.func2(0xa93c8f, 0xc0005d5540)
	/app/pkg/kube/object_patch/patch.go:129 +0x3fa
k8s.io/client-go/util/retry.OnError.func1(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:51 +0x3c
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0xc0008aee18, 0x100, 0x0, 0x0)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:211 +0x69
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0, 0xc0008aee18, 0x0, 0x0)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:399 +0x55
k8s.io/client-go/util/retry.OnError(0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0, 0x1a881d0, 0xc0008af050, 0x0, 0x0)
	/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:50 +0xa6
k8s.io/client-go/util/retry.RetryOnConflict(...)
	/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:104
github.com/flant/shell-operator/pkg/kube/object_patch.(*ObjectPatcher).executeCreateOperation(0xc0002c6978, 0xc000720840, 0x1bc4601, 0xc00070f030)
	/app/pkg/kube/object_patch/patch.go:118 +0x745
github.com/flant/shell-operator/pkg/kube/object_patch.(*ObjectPatcher).ExecuteOperation(0xc0002c6978, 0x1bc38a0, 0xc000720840, 0x16, 0xc0008af170)
	/app/pkg/kube/object_patch/patch.go:65 +0x130
github.com/flant/shell-operator/pkg/kube/object_patch.(*ObjectPatcher).ExecuteOperations(0xc0002c6978, 0xc0007d9100, 0x6, 0x8, 0x0, 0x0)
	/app/pkg/kube/object_patch/patch.go:49 +0x236
github.com/flant/shell-operator/pkg/shell-operator.(*ShellOperator).HandleRunHook(0xc0002bd520, 0x1c12f68, 0xc000098280, 0xc000457cc0, 0xc0002cf4e4, 0x11, 0xc0006f0b10, 0xf, 0x0, 0x0, ...)
	/app/pkg/shell-operator/operator.go:761 +0x485
github.com/flant/shell-operator/pkg/shell-operator.(*ShellOperator).TaskHandleHookRun(0xc0002bd520, 0x1c12f68, 0xc000098280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/app/pkg/shell-operator/operator.go:708 +0xbe8
github.com/flant/shell-operator/pkg/shell-operator.(*ShellOperator).TaskHandler(0xc0002bd520, 0x1c12f68, 0xc000098280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/app/pkg/shell-operator/operator.go:538 +0x1f0
github.com/flant/shell-operator/pkg/task/queue.(*TaskQueue).Start.func1(0xc000698420)
	/app/pkg/task/queue/task_queue.go:342 +0x3fc
created by github.com/flant/shell-operator/pkg/task/queue.(*TaskQueue).Start
	/app/pkg/task/queue/task_queue.go:320 +0x4f

As it crashes and brings down the pod, it restarts, but as the updates couldn't be applied to the cluster, after restart the same action gets triggered again via Synchronization rather than Event, and crashes once more.

Environment:

Shell-operator version: flant/shell-operator:v1.0.7

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1+k3s2", GitCommit:"a0cadcd3436745396c82821b4a0803ee460401e1", GitTreeState:"clean", BuildDate:"2022-01-14T08:22:21Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
  • Installation type (kubectl apply, helm chart, etc.):

Custom deployment using kapp.

Anything else we should know?:

I will update with more details in comments when narrow down which specific Kubernetes resource type is triggering the issue.

GrahamDumpleton avatar Jan 27 '22 01:01 GrahamDumpleton

Seems to be enough to output any resource which contains a property with an integer field and write it to KUBERNETES_PATCH_PATH. Eg.,

apiVersion: v1
kind: Service
metadata:
  name: xxx
  namespace: default
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 5000
      targetPort: 5000
      protocol: TCP
  selector:
    app: xxx

Also fails with an Ingress and Deployment, thus presumably also resources such as StatefulSet and DaemonSet where ports are specified.

Previously I had only been creating Secret resources using Shell-operator so hadn't encountered the problem. This issue sort of makes KUBERNETES_PATCH_PATH unusable for more complex use cases. :-(

GrahamDumpleton avatar Jan 27 '22 02:01 GrahamDumpleton

Possibly relevant issue:

  • https://github.com/kubernetes/kubernetes/issues/62769

GrahamDumpleton avatar Jan 27 '22 02:01 GrahamDumpleton