opnsense
Kernel Panic: general protection fault in ng_netflow_flow6_add
Important notices
Before you add a new report, we ask you kindly to acknowledge the following:
- [X] I have read the contributing guide lines at https://github.com/opnsense/src/blob/master/CONTRIBUTING.md
- [X] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/src/issues?q=is%3Aissue
Describe the bug
The system experiences a recurring kernel panic when the NetFlow / Insight reporting feature is enabled. The panic is a "Fatal trap 9: general protection fault" that occurs deep within the kernel's networking stack, specifically within the Netgraph NetFlow module.
The backtrace consistently points to the ng_netflow_flow6_add function, indicating the crash is triggered when processing an IPv6 data flow.
To Reproduce
Steps to reproduce the behavior: Enable the "Insight" service under Reporting -> Insight on one or more interfaces that handle IPv6 traffic.
Allow network traffic to flow normally.
The system will eventually panic and reboot. The trigger appears to be a specific type of IPv6 packet, but it occurs during regular operation.
Expected behavior
The system should remain stable with the NetFlow/Insight service enabled, correctly processing both IPv4 and IPv6 traffic without causing a kernel panic.
Relevant log files
Crash report
[639965] load_dn_sched dn_sched FIFO loaded
[639965] load_dn_sched dn_sched QFQ loaded
[639965] load_dn_sched dn_sched RR loaded
[639965] load_dn_sched dn_sched WF2Q+ loaded
[639965] load_dn_sched dn_sched PRIO loaded
[639965] load_dn_sched dn_sched FQ_CODEL loaded
[639965] load_dn_sched dn_sched FQ_PIE loaded
[639965] load_dn_aqm dn_aqm CODEL loaded
[639965] load_dn_aqm dn_aqm PIE loaded
[640128] ipfw2 (+ipv6) initialized, divert loadable, nat loadable, default to accept, logging disabled
[833564]
[833564]
[833564] Fatal trap 9: general protection fault while in kernel mode
[833564] cpuid = 15; apic id = 2e
[833564] instruction pointer = 0x20:0xffffffff80bb48e9
[833564] stack pointer = 0x28:0xfffffe0038f4c940
[833564] frame pointer = 0x28:0xfffffe0038f4c9c0
[833564] code segment = base 0x0, limit 0xfffff, type 0x1b
[833564] = DPL 0, pres 1, long 1, def32 0, gran 1
[833564] processor eflags = interrupt enabled, resume, IOPL = 0
[833564] current process = 0 (if_io_tqg_15)
[833564] rdi: fffffe01888c2368 rsi: 0040000000000000 rdx: 0000000000000000
[833564] rcx: 0000000000000001 r8: 0000000006000003 r9: 0000000000000000
[833564] rax: 0040000000000000 rbx: fffff80001b93000 rbp: fffffe0038f4c9c0
[833564] r10: 0000000000000050 r11: 0000000000000002 r12: 0000000000000000
[833564] r13: fffff80001b93000 r14: fffffe0038f4c968 r15: fffffe01888c2368
[833564] trap number = 9
[833564] panic: general protection fault
[833564] cpuid = 15
[833564] time = 1755876649
[833564] KDB: stack backtrace:
[833564] db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0038f4c6c0
[833564] vpanic() at vpanic+0x161/frame 0xfffffe0038f4c7f0
[833564] panic() at panic+0x43/frame 0xfffffe0038f4c850
[833564] trap_fatal() at trap_fatal+0x68/frame 0xfffffe0038f4c870
[833564] calltrap() at calltrap+0x8/frame 0xfffffe0038f4c870
[833564] --- trap 0x9, rip = 0xffffffff80bb48e9, rsp = 0xfffffe0038f4c940, rbp = 0xfffffe0038f4c9c0 ---
[833564] __mtx_lock_sleep() at __mtx_lock_sleep+0xc9/frame 0xfffffe0038f4c9c0
[833564] ng_netflow_flow6_add() at ng_netflow_flow6_add+0x50e/frame 0xfffffe0038f4ca70
[833564] ng_netflow_rcvdata() at ng_netflow_rcvdata+0x83f/frame 0xfffffe0038f4caf0
[833564] ng_apply_item() at ng_apply_item+0x147/frame 0xfffffe0038f4cb90
[833564] ng_snd_item() at ng_snd_item+0x26c/frame 0xfffffe0038f4cbd0
[833564] ng_ether_input() at ng_ether_input+0x4c/frame 0xfffffe0038f4cc00
[833564] ether_nh_input() at ether_nh_input+0x1dc/frame 0xfffffe0038f4cc60
[833564] netisr_dispatch_src() at netisr_dispatch_src+0x9f/frame 0xfffffe0038f4ccb0
[833564] ether_input() at ether_input+0x56/frame 0xfffffe0038f4cd00
[833564] iflib_rxeof() at iflib_rxeof+0xc4e/frame 0xfffffe0038f4ce00
[833564] _task_fn_rx() at _task_fn_rx+0x72/frame 0xfffffe0038f4ce40
[833564] gtaskqueue_run_locked() at gtaskqueue_run_locked+0x14e/frame 0xfffffe0038f4cec0
[833564] gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc2/frame 0xfffffe0038f4cef0
[833564] fork_exit() at fork_exit+0x81/frame 0xfffffe0038f4cf30
[833564] fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0038f4cf30
[833564] --- trap 0x8054bef0, rip = 0x1ec5e32, rsp = 0, rbp = 0xc ---
[833564] KDB: enter: panic
Environment
Software version used and hardware type if relevant, e.g.:
OPNsense Version: 25.7.1_1 FreeBSD Version: 14.3-RELEASE-p1 Architecture: amd64 (SMP)
Hi, happened again. Seems that this happen every 2-3 weeks randomly.
[1060455]
[1060455]
[1060455] Fatal trap 9: general protection fault while in kernel mode
[1060455] cpuid = 4; apic id = 10
[1060455] instruction pointer = 0x20:0xffffffff80bb48e9
[1060455] stack pointer = 0x28:0xfffffe00e33fe620
[1060455] frame pointer = 0x28:0xfffffe00e33fe6a0
[1060455] code segment = base 0x0, limit 0xfffff, type 0x1b
[1060455] = DPL 0, pres 1, long 1, def32 0, gran 1
[1060455] processor eflags = interrupt enabled, resume, IOPL = 0
[1060455] current process = 0 (if_io_tqg_4)
[1060455] rdi: fffffe018811b128 rsi: 0008000000000000 rdx: 0000000000000000
[1060455] rcx: 0000000000000001 r8: 0000000000003903 r9: 0000000000000000
[1060455] rax: 0008000000000000 rbx: fffffe0187d10000 rbp: fffffe00e33fe6a0
[1060455] r10: 00000000a42241f4 r11: 0000000000000000 r12: 0000000000000000
[1060455] r13: fffff80001aa6740 r14: fffffe00e33fe648 r15: fffffe018811b128
[1060455] trap number = 9
[1060455] panic: general protection fault
[1060455] cpuid = 4
[1060455] time = 1756937198
[1060455] KDB: stack backtrace:
[1060455] db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00e33fe3a0
[1060455] vpanic() at vpanic+0x161/frame 0xfffffe00e33fe4d0
[1060455] panic() at panic+0x43/frame 0xfffffe00e33fe530
[1060455] trap_fatal() at trap_fatal+0x68/frame 0xfffffe00e33fe550
[1060455] calltrap() at calltrap+0x8/frame 0xfffffe00e33fe550
[1060455] --- trap 0x9, rip = 0xffffffff80bb48e9, rsp = 0xfffffe00e33fe620, rbp = 0xfffffe00e33fe6a0 ---
[1060455] __mtx_lock_sleep() at __mtx_lock_sleep+0xc9/frame 0xfffffe00e33fe6a0
[1060455] ng_netflow_flow_add() at ng_netflow_flow_add+0x502/frame 0xfffffe00e33fe730
[1060455] ng_netflow_rcvdata() at ng_netflow_rcvdata+0x7cb/frame 0xfffffe00e33fe7b0
[1060455] ng_apply_item() at ng_apply_item+0x147/frame 0xfffffe00e33fe850
[1060455] ng_snd_item() at ng_snd_item+0x26c/frame 0xfffffe00e33fe890
[1060455] ng_ether_input() at ng_ether_input+0x4c/frame 0xfffffe00e33fe8c0
[1060455] ether_nh_input() at ether_nh_input+0x1dc/frame 0xfffffe00e33fe920
[1060455] netisr_dispatch_src() at netisr_dispatch_src+0x9f/frame 0xfffffe00e33fe970
[1060455] ether_input() at ether_input+0x56/frame 0xfffffe00e33fe9c0
[1060455] ether_demux() at ether_demux+0x8e/frame 0xfffffe00e33fe9f0
[1060455] ng_ether_rcv_upper() at ng_ether_rcv_upper+0x8c/frame 0xfffffe00e33fea10
[1060455] ng_apply_item() at ng_apply_item+0x147/frame 0xfffffe00e33feab0
[1060455] ng_snd_item() at ng_snd_item+0x26c/frame 0xfffffe00e33feaf0
[1060455] ng_apply_item() at ng_apply_item+0x147/frame 0xfffffe00e33feb90
[1060455] ng_snd_item() at ng_snd_item+0x26c/frame 0xfffffe00e33febd0
[1060455] ng_ether_input() at ng_ether_input+0x4c/frame 0xfffffe00e33fec00
[1060455] ether_nh_input() at ether_nh_input+0x1dc/frame 0xfffffe00e33fec60
[1060455] netisr_dispatch_src() at netisr_dispatch_src+0x9f/frame 0xfffffe00e33fecb0
[1060455] ether_input() at ether_input+0x56/frame 0xfffffe00e33fed00
[1060455] iflib_rxeof() at iflib_rxeof+0xc4e/frame 0xfffffe00e33fee00
[1060455] _task_fn_rx() at _task_fn_rx+0x72/frame 0xfffffe00e33fee40
[1060455] gtaskqueue_run_locked() at gtaskqueue_run_locked+0x14e/frame 0xfffffe00e33feec0
[1060455] gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc2/frame 0xfffffe00e33feef0
[1060455] fork_exit() at fork_exit+0x81/frame 0xfffffe00e33fef30
[1060455] fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e33fef30
[1060455] --- trap 0x223d6449, rip = 0x3833383038326634, rsp = 0x61702c686374616d, rbp = 0x61632e77666f2030 ---
[1060455] KDB: enter: panic
panic.txt0600003015056135756 7146 ustarrootwheelgeneral protection faultversion.txt0600007515056135756 7552 ustarrootwheelFreeBSD 14.3-RELEASE-p1 stable/25.7-n271606-9af17f0102ca SMP
@fichtner @AdSchellevis
Can be fixed in the next release? Thanks.
I concur your issue is the netflow capture. But since this is FreeBSD kernel code that’s likely not even maintained the best suggestion is to disable netflow.
Hi @fichtner , thanks for your reply. I set the netflow as follow
There is a plugin can do the same thing or similar? Thanks. :)
There is a newer way to do this in upcoming FreeBSD 15 but it's not available in any 14.x release. While 15.0 should be available at the end of the year we will likely aim for 15.1 to be safe.
As for issues with ng_netflow_flow(6)_add() locking I've checked the source code and there is no apparent locking fix in the current netgraph code anywhere which basically means nobody else is using this extensively enough to hit the bug. No FreeBSD report, no interested maintainer, no bug fix.
Cheers, Franco
Thank you very much @fichtner . I really appreciate your work. I'll wait, if there is not a plugin can do the same things. Thanks!
Gennaro