High memory consumption

[stefankuksenko]

Eventrouter is constantly increasing the amount of memory consumed. In one of our cluster after one month eventrouter uses near 8Gb of memory.

It is the output of top command in eventrouter pod (containter)

  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
    6     1 nobody   S    7944m  23%   1   0% /eventrouter -v 3 -logtostderr
   32     0 nobody   S     1524   0%   6   0% sh
   43    32 nobody   R     1520   0%   0   0% top
    1     0 nobody   S     1516   0%   5   0% /bin/sh -c /eventrouter -v 3 -logtostderr

I will configure limits. But what will happen if eventrouter reaches the memory limit? Will it work correctly. And why does it use so much memory?

[timothysc]

So something doesn't seem right.
@johnSchnake

[stefankuksenko]

still have this problem in all our clusters in GKE. When i limit memory usage for eventrouter pod memory usage increases during a time, when memory rests to limit pod restarts with "OOMKilled" reason. There are 247 restarts during seven days.

    State:          Running
      Started:      Thu, 23 Jan 2020 13:00:54 +0300
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Thu, 23 Jan 2020 12:48:53 +0300
      Finished:     Thu, 23 Jan 2020 13:00:51 +0300
    Ready:          True
    Restart Count:  247
    Limits:
      cpu:     100m
      memory:  256Mi
    Requests:
      cpu:        100m
      memory:     256Mi

[marckamerbeek]

Same problem here...

[njlkj]

involved_object_name is a label with high cardinality

kubernetesWarningEventCounterVec = prometheus.NewCounterVec(prometheus.CounterOpts{
	Name: "heptio_eventrouter_warnings_total",
	Help: "Total number of warning events in the kubernetes cluster",
}, []string{
	"involved_object_kind",
	"involved_object_name",
	"involved_object_namespace",
	"reason",
	"source",
})

[njlkj]

So something doesn't seem right. @johnSchnake

#119