Node-Exporter : memory usage too high (OOME)

[SpeedBlack]

Host operating system: output of uname -a

3.10.0-862.3.2.el7.x86_64

node_exporter version: output of node_exporter --version

sh-4.2$ node_exporter --version
node_exporter, version 0.16.0 (branch: HEAD, revision: d42bd70f4363dced6b77d8fc311ea57b63387e4f)
build user: root@node-exporter-binary-3-build
build date: 20180606-16:48:15
go version: go1.10

node_exporter command line flags

--path.procfs=/host/proc --path.sysfs=/host/sys

Are you running node_exporter in Docker?

Yes, in Openshift

Hi,

I use node-exporter (openshift/prometheus-node-exporter:v0.16.0) in Openshift with Prometheus and Grafana. I have a problem with memory recycling. The Pod is killed each time after at an OOME (OutOfMemory). Memory usage increases continuously without being recycled.

By default, the limits were (template here):

          resources:
            requests:
              memory: 30Mi
              cpu: 100m
            limits:
              memory: 50Mi
              cpu: 200m

I tested several configurations without success. Today, the configuration is:

          resources:
            limits:
              cpu: 250m
              memory: 250Mi
            requests:
              cpu: 100m
              memory: 75Mi

Do you have any idea ? An adjustment to make ? Do you have recommended resources limits ?

Thanks in advance for your help !

[SuperQ]

The easiest thing to do is get a pprof memory profile.

See https://github.com/prometheus/node_exporter/issues/859 for some more tips.

It would be useful to look at the the node_exporter's metric, go_goroutines over time to see if there is a leak.

[SpeedBlack]

Hi SuperQ,

Below, the memory usage of the "node-exporter" pod deployed on a node :

below the graph with the metrics go_goroutines for the same node : go_goroutines{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="xxxxxx",job="kubernetes-nodes-exporter",kubernetes_io_hostname="xxxxxx",.....}

What is this metric ?

I'm looking at how to get the pprof.

Thanks, Best regards.

[SuperQ]

Yes, that looks like a goroutine leak. Can you post any pod logs to see if there are any errors?

[SpeedBlack]

Below the pod logs :

$ oc logs -f prometheus-node-exporter-krvrb
time="2018-07-19T03:57:03Z" level=info msg="Starting node_exporter (version=0.16.0, branch=HEAD, revision=d42bd70f4363dced6b77d8fc311ea57b63387e4f)" source="node_exporter.go:82"
time="2018-07-19T03:57:03Z" level=info msg="Build context (go=go1.10, user=root@node-exporter-binary-3-build, date=20180606-16:48:15)" source="node_exporter.go:83"
time="2018-07-19T03:57:03Z" level=info msg="Enabled collectors:" source="node_exporter.go:90"
time="2018-07-19T03:57:03Z" level=info msg=" - arp" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - bcache" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - bonding" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - conntrack" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - cpu" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - diskstats" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - edac" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - entropy" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - filefd" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - filesystem" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - hwmon" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - infiniband" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - ipvs" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - loadavg" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - mdadm" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - meminfo" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - netdev" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - netstat" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - nfs" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - nfsd" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - sockstat" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - stat" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - textfile" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - time" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - timex" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - uname" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - vmstat" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - wifi" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - xfs" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg=" - zfs" source="node_exporter.go:97"
time="2018-07-19T03:57:03Z" level=info msg="Listening on :9100" source="node_exporter.go:111"

Thanks

[SuperQ]

Thanks, hopefully the pprof dump will give us the information we need.

[SpeedBlack]

Hi Superq,

I could run the command but Graphviz is not installed and I can't install it in the pod:

go tool pprof -svg http://localhost:9090/debug/pprof/heap > heap.svg                                                                                                                      
Fetching profile over HTTP from http://localhost:9090/debug/pprof/heap                                                                                                                            
Could not use temp dir /pprof: mkdir /pprof: permission denied                                                                                                                                    
Saved profile in /tmp/pprof.prometheus.alloc_objects.alloc_space.inuse_objects.inuse_space.003.pb.gz                                                                                              
Failed to execute dot. Is Graphviz installed? Error: exec: "dot": executable file not found in $PATH

I have to run the command in the pod because I have an OCP auth-proxy for prometheus.

But I have the raw data ("pprof.prometheus.alloc_objects.alloc_space.inuse_objects.inuse_space.003.pb.gz" file). Do you know how I can generate the graph afterwards with Graphviz on another system ? I haven't found the command yet... or the tool.

Thanks, Best regards.

[SpeedBlack]

Sorry, it's Ok :

it's Ok ?

Thanks, Best regards

[SuperQ]

That is a profile for Prometheus, not the node exporter. You need to use port 9100.

[SpeedBlack]

Hoo sorry, Please, find attachment pprof svg for node-exporter.

pprof_node_exporter.zip

Thanks Best regards

[SpeedBlack]

Below, the memory usage of the node-exporter since the restart of the pod:

[SuperQ]

It would help to take a pprof sample when it's using over 200MB of memory, this way we can more easily find the memory leak.

[SpeedBlack]

I took a pprof sample at 12:24:00. The node-exporter used 248MiB :

is it better to lower the MEMORY limit and take a pprof sample ? (Currently 250Mi | By default 50Mi)

Thanks, Best regards.

[SuperQ]

Strange. The pprof only shows 9.6MiB fo memory in use.

I would probably eliminate the CPU limit. I have seen CPU limits in K8s cause bad performance stalling. But I haven't looked at the current implementation to see if that's improved or not.

/cc @juliusv Do you know if there is something specific we can do to trace this goroutine leak?

[SpeedBlack]

Hi, I tried to remove CPU limits without result. I'll take another pprof sample.

Thanks, Best regards

[SuperQ]

One additional debugging option, get a dump of the goroutines with http://localhost:9100/debug/pprof/goroutine?debug=1. Normally it would be only a few, like this.

[juliusv]

@SuperQ Yeah, I would recommend listing the active goroutines, as we already know it's connected to a goroutine leak. But prefer debug=2 over debug=1, so: http://localhost:9100/debug/pprof/goroutine?debug=2

[SpeedBlack]

Hi,

I can't execute the command with debug=2 option :

# go tool pprof -svg http://xxxxx:9100/debug/pprof/goroutine?debug=2 > dump_goroutine_2.svg
Fetching profile over HTTP from http://xxxxx:9100/debug/pprof/goroutine?debug=2
http://xxxxx:9100/debug/pprof/goroutine?debug=2: parsing profile: unrecognized profile format
failed to fetch any profiles

Do you have any idea?

For the moment, please, find attachment dump of the goroutines (debug=1 option) and new pprof sample heap for node-exporter. dumps.zip

For information, currently, the pod node-exporter uses 137MiB

Thanks, Best regards.

[SuperQ]

The goroutine link doesn't require pprof, just curl. It's just a big text dump, not svg.

[SpeedBlack]

Ok, thanks for the clarification.

Please, find attachment dump of the goroutines (debug=1 & debug=2). goroutine_dump.zip

Thanks, Best regards.

[SuperQ]

Looks like it's getting stuck on netlink. This is used by the wifi collector, try running the node_exporter with --no-collector.wifi

/cc @mdlayher 😁

goroutine 17790 [chan receive, 501 minutes, locked to thread]:
github.com/prometheus/node_exporter/vendor/github.com/mdlayher/netlink.newSysSocket.func1(0xc420570e01, 0xc4206bc2f0, 0xc420570e40)
        /go/src/github.com/prometheus/node_exporter/vendor/github.com/mdlayher/netlink/conn_linux.go:258 +0x8a
created by github.com/prometheus/node_exporter/vendor/github.com/mdlayher/netlink.newSysSocket
        /go/src/github.com/prometheus/node_exporter/vendor/github.com/mdlayher/netlink/conn_linux.go:241 +0xa0

[simonpasquier]

@SpeedBlack also as you're using OpenShift, please note that the wifi collector should now be disabled by default in Ansible-based installations.

[mdlayher]

Interesting. It appears to be stuck in the locked syscall goroutine I have set up. I probably don't have time to look at this immediately, but can OP please clarify if they have WiFi devices in this system?

[SpeedBlack]

Great ! Thanks for your help ! I just added "--no-collector.wifi". I'll confirm tomorrow if the problem is gone !

@mdlayher, no I don't have wifi devices in this system.

Thanks, Best regards

[SpeedBlack]

Hi,

Today, I confirm it's Ok ! So, I close this case !

Thanks for your help ! Best regards.

[mdlayher]

I'm leaving this open for tracking.

[SuperQ]

@mdlayher Maybe we should file a new issue with a cleaned up summary.

[SuperQ]

@mdlayher Did we ever get the upstream issue solved for this?

[mdlayher]

I don't believe so. I haven't touched that code in quite a while and wasn't able to easily reproduce these results.

[alter]

The same, hello from 2021

[ovidiubuligan]

Sorry, it's Ok :

it's Ok ?

Thanks, Best regards

Hello , you have an interesting chart there . How did you generate that map ?