carina icon indicating copy to clipboard operation
carina copied to clipboard

throw some exception when kubelet terminate csi node container

Open fanhaouu opened this issue 2 years ago • 1 comments

What happened: {"level":"info","ts":1659103499.6545281,"msg":"Stopping and waiting for non leader election runnables"} {"level":"info","ts":1659103529.6545928,"msg":"Stopping and waiting for leader election runnables"} {"level":"info","ts":1659103529.6546292,"msg":"Stopping and waiting for caches"} {"level":"info","ts":1659103529.654654,"msg":"Stopping and waiting for webhooks"} {"level":"info","ts":1659103529.6546695,"msg":"Wait completed, proceeding to shutdown the manager"} {"level":"error","ts":1659103529.654631,"logger":"setup","msg":"problem running manager","error":"failed waiting for all runnables to end within grace period of 30s: context deadline exceeded","stacktrace":"github.com/carina-io/carina/cmd/carina-node/run.subMain\n\t/workspace/github.com/carina-io/carina/cmd/carina-node/run/run.go:163\ngithub.com/carina-io/carina/cmd/carina-node/run.glob..func1\n\t/workspace/github.com/carina-io/carina/cmd/carina-node/run/root.go:48\ngithub.com/spf13/cobra.(*Command).execute\n\t/workspace/github.com/carina-io/carina/vendor/github.com/spf13/cobra/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/workspace/github.com/carina-io/carina/vendor/github.com/spf13/cobra/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\t/workspace/github.com/carina-io/carina/vendor/github.com/spf13/cobra/command.go:902\ngithub.com/carina-io/carina/cmd/carina-node/run.Execute\n\t/workspace/github.com/carina-io/carina/cmd/carina-node/run/root.go:55\nmain.main\n\t/workspace/github.com/carina-io/carina/cmd/carina-node/main.go:29\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255"} {"level":"info","ts":1659103529.654709,"logger":"controller.nodestorageresource","msg":"Shutdown signal received, waiting for all workers to finish","reconciler group":"carina.storage.io","reconciler kind":"NodeStorageResource"} panic: close of closed channel

goroutine 1 [running]: github.com/carina-io/carina/cmd/carina-node/run.subMain() /workspace/github.com/carina-io/carina/cmd/carina-node/run/run.go:165 +0xb88 github.com/carina-io/carina/cmd/carina-node/run.glob..func1(0x28b9440, {0x1920d0d, 0x3, 0x3}) /workspace/github.com/carina-io/carina/cmd/carina-node/run/root.go:48 +0x1e github.com/spf13/cobra.(*Command).execute(0x28b9440, {0xc000138050, 0x3, 0x3}) /workspace/github.com/carina-io/carina/vendor/github.com/spf13/cobra/command.go:856 +0x60e github.com/spf13/cobra.(*Command).ExecuteC(0x28b9440) /workspace/github.com/carina-io/carina/vendor/github.com/spf13/cobra/command.go:974 +0x3bc github.com/spf13/cobra.(*Command).Execute(...) /workspace/github.com/carina-io/carina/vendor/github.com/spf13/cobra/command.go:902 github.com/carina-io/carina/cmd/carina-node/run.Execute() /workspace/github.com/carina-io/carina/cmd/carina-node/run/root.go:55 +0x25 main.main() /workspace/github.com/carina-io/carina/cmd/carina-node/main.go:29 +0x1c

What you expected to happen: no any exception

How to reproduce it: kubectl delete csinode-pod -n carina

fanhaouu avatar Jul 29 '22 14:07 fanhaouu

I will take a PR to fix this bug.

fanhaouu avatar Jul 29 '22 14:07 fanhaouu