constantly crashing in 1 cluster

[mtougeron]

I'm getting the following error in one of my Kubernetes clusters. It works fine in other clusters of the same version & configuration. I have no idea how to reproduce this; it just keeps happening in this one cluster. It happened when my cluster was running k8s v1.13.7 and still happens after upgrading to v1.14.6. All other clusters of the same k8s versions works fine.

Does anyone have any suggestions for how to resolve this?

Container: gcr.io/heptio-images/eventrouter:v0.2 Kubernetes: v1.14.6

E0820 17:52:52.754409       6 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/src/github.com/heptiolabs/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/go/src/github.com/heptiolabs/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/heptiolabs/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:509
/usr/local/go/src/runtime/panic.go:491
/usr/local/go/src/runtime/panic.go:63
/usr/local/go/src/runtime/signal_unix.go:367
/go/src/github.com/heptiolabs/eventrouter/eventrouter.go:152
/go/src/github.com/heptiolabs/eventrouter/eventrouter.go:113
/go/src/github.com/heptiolabs/eventrouter/eventrouter.go:86
/go/src/github.com/heptiolabs/eventrouter/vendor/k8s.io/client-go/tools/cache/controller.go:195
<autogenerated>:1
/go/src/github.com/heptiolabs/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:545
/go/src/github.com/heptiolabs/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:381
/go/src/github.com/heptiolabs/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
/usr/local/go/src/runtime/asm_amd64.s:2337
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x119a173]

goroutine 91 [running]:
github.com/heptiolabs/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/heptiolabs/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x111
panic(0x1300660, 0x1e748a0)
	/usr/local/go/src/runtime/panic.go:491 +0x283
main.prometheusEvent(0xc420e8d200)
	/go/src/github.com/heptiolabs/eventrouter/eventrouter.go:152 +0xf3
main.(*EventRouter).addEvent(0xc4200ca4c0, 0x147bf80, 0xc420e8d200)
	/go/src/github.com/heptiolabs/eventrouter/eventrouter.go:113 +0x3c
main.(*EventRouter).(main.addEvent)-fm(0x147bf80, 0xc420e8d200)
	/go/src/github.com/heptiolabs/eventrouter/eventrouter.go:86 +0x3e
github.com/heptiolabs/eventrouter/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(0xc4202107b0, 0xc4202107c0, 0xc4202107d0, 0x147bf80, 0xc420e8d200)
	/go/src/github.com/heptiolabs/eventrouter/vendor/k8s.io/client-go/tools/cache/controller.go:195 +0x49
github.com/heptiolabs/eventrouter/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnAdd(0xc4201a6e20, 0x147bf80, 0xc420e8d200)
	<autogenerated>:1 +0x62
github.com/heptiolabs/eventrouter/vendor/k8s.io/client-go/tools/cache.(*processorListener).run(0xc42041a780)
	/go/src/github.com/heptiolabs/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:545 +0x272
github.com/heptiolabs/eventrouter/vendor/k8s.io/client-go/tools/cache.(*processorListener).(github.com/heptiolabs/eventrouter/vendor/k8s.io/client-go/tools/cache.run)-fm()
	/go/src/github.com/heptiolabs/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:381 +0x2a
github.com/heptiolabs/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc4201d4ad8, 0xc420210880)
	/go/src/github.com/heptiolabs/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x4f
created by github.com/heptiolabs/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/src/github.com/heptiolabs/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62

[simonkrenger]

I think you are hitting the same issue that we observe here: Red Hat BugZilla #1701495

We can reproduce the issue by creating Events that are not of "type": "Normal" or "type": "Warning". The issue is that most likely, counter is not set when the "type" is set to something else here: https://github.com/heptiolabs/eventrouter/blob/master/eventrouter.go#L152

[mtougeron]

That could be it then. We have an event type of StateChange for a CRD

[jselleck]

FYI I ran into this while using Argo Events on the same cluster. In my case LAST SEEN was , TYPE was set to one of ["OperationSuccessful"."StateChange"], reason was a short bit of descriptive text, The object on the event for these was sensor/, and Message was empty.

Since we're not tracking with prometheous I set "enable-prometheus": false in the config.json

It would be nice if this could be handled.

Thanks for the great project!

[KouriR]

This looks like it was fixed via https://github.com/openshift/eventrouter/pull/11.