panic: runtime error: invalid memory address or nil pointer dereference

[agt-the-walker]

Bug Overview

Since installing nginx-agent-2.40.0-1.x86_64 RPM on our RHEL 9.5 systems, we keep getting the following errors:

$ sudo journalctl | fgrep -B2 -A20 -w 'panic: runtime error: invalid memory address or nil pointer dereference' | tail -23
Apr 02 11:44:20 redacted.unil.ch systemd[1]: Stopped User Runtime Directory /run/user/0.
Apr 02 11:44:20 redacted.unil.ch systemd[1]: Removed slice User Slice of UID 0.
Apr 02 11:44:34 redacted.unil.ch nginx-agent[956060]: panic: runtime error: invalid memory address or nil pointer dereference
Apr 02 11:44:34 redacted.unil.ch nginx-agent[956060]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xb96086]
Apr 02 11:44:34 redacted.unil.ch nginx-agent[956060]: goroutine 3720 [running]:
Apr 02 11:44:34 redacted.unil.ch nginx-agent[956060]: github.com/nginx/agent/v2/src/core/metrics.GenerateMetricsReportBundle({0xc002403f68, 0x1, 0x1?})
Apr 02 11:44:34 redacted.unil.ch nginx-agent[956060]:         /home/runner/work/agent/agent/src/core/metrics/metrics_util.go:405 +0xc6
Apr 02 11:44:34 redacted.unil.ch nginx-agent[956060]: github.com/nginx/agent/v2/src/plugins.(*Metrics).drainBuffer(0xc0002843c0, {0x1021660, 0xc0000b2f50})
Apr 02 11:44:34 redacted.unil.ch nginx-agent[956060]:         /home/runner/work/agent/agent/src/plugins/metrics.go:208 +0xea
Apr 02 11:44:34 redacted.unil.ch nginx-agent[956060]: created by github.com/nginx/agent/v2/src/plugins.(*Metrics).Init in goroutine 1
Apr 02 11:44:34 redacted.unil.ch nginx-agent[956060]:         /home/runner/work/agent/agent/src/plugins/metrics.go:70 +0x166
Apr 02 11:44:34 redacted.unil.ch systemd[1]: nginx-agent.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 02 11:44:34 redacted.unil.ch systemd[1]: nginx-agent.service: Failed with result 'exit-code'.
Apr 02 11:44:34 redacted.unil.ch systemd[1]: nginx-agent.service: Consumed 1min 20.489s CPU time.
Apr 02 11:44:39 redacted.unil.ch systemd[1]: nginx-agent.service: Scheduled restart job, restart counter is at 79.
Apr 02 11:44:39 redacted.unil.ch systemd[1]: Stopped NGINX Agent.
Apr 02 11:44:39 redacted.unil.ch systemd[1]: nginx-agent.service: Consumed 1min 20.489s CPU time.
Apr 02 11:44:39 redacted.unil.ch systemd[1]: Starting NGINX Agent...
Apr 02 11:44:39 redacted.unil.ch systemd[1]: Started NGINX Agent.
Apr 02 11:44:39 redacted.unil.ch nginx-agent[962904]: time="2025-04-02T11:44:39+02:00" level=warning msg="Log level is info"
Apr 02 11:44:39 redacted.unil.ch nginx-agent[962904]: time="2025-04-02T11:44:39+02:00" level=info msg="setting displayName to redacted.unil.ch"
Apr 02 11:44:39 redacted.unil.ch nginx-agent[962904]: time="2025-04-02T11:44:39+02:00" level=info msg="NGINX Agent v2.40.0 at 93c4603d with pid 962904, clientID=9[...]9 name=redacted.unil.ch features=[registration nginx-config-async nginx-ssl-config nginx-counting metrics dataplane-status process-watcher file-watcher activity-events agent-api]"
Apr 02 11:44:39 redacted.unil.ch nginx-agent[962904]: time="2025-04-02T11:44:39+02:00" level=warning msg="Virtualization not set, defaulting to host.VirtualizationWithContext"

Expected Behavior

No "panic: runtime error: invalid memory address or nil pointer dereference" or "SIGSEGV".

Steps to Reproduce the Bug

Enable/start nginx-agentservice. This issue occurs randomly:

$ sudo journalctl | fgrep 'panic: runtime error: invalid memory address or nil pointer dereference' | cut -c1-9 | sort -M | uniq -c
      1 Mar 31 02
      2 Mar 31 05
      1 Mar 31 06
      1 Mar 31 07
      1 Mar 31 15
      2 Mar 31 18
      1 Mar 31 19
      1 Mar 31 20
      1 Mar 31 23
      1 Apr 01 00
      1 Apr 01 01
      1 Apr 01 02
      1 Apr 01 05
      1 Apr 01 06
      1 Apr 01 09
      1 Apr 01 11
      3 Apr 01 12
      2 Apr 01 13
      1 Apr 01 14
      1 Apr 01 15
      1 Apr 01 19
      1 Apr 01 20
      2 Apr 01 21
      1 Apr 02 00
      1 Apr 02 01
      2 Apr 02 03
      2 Apr 02 06
      1 Apr 02 07
      1 Apr 02 08
      2 Apr 02 09
      1 Apr 02 10
      1 Apr 02 11

Environment Details

• Target deployment platform: local cluster
• Target OS: RHEL 9.5
• Version of this project or specific commit: 2.40.0-1.x86_64

Additional Context

We have a subscription to "NGINX One (per instance) support Premium".

More details upon request.

Jérôme Zago, UNIL.

[nginx-seanmoloney]

Hi @agt-the-walker ,

Thanks for opening the issue, Would it be possible to get the nginx-agent.conf and nginx.conf files.

Also can I ask that you use F5 Support to create NGINX One support cases.

Thanks, Sean

[agt-the-walker]

Thanks for your quick reply. Created case 00832909.

[Samseppiol]

Also experiencing this issue

[Samseppiol]

For anyone else experiencing this issue, our nginx instance manager was out of disk space, therefore the agent couldn't connect.

[nginx-seanmoloney]

Hi @Samseppiol , how are you?

This issue has been fixed in v2.40.1

Ty