opnsense
24.7 - Kernel panic when pinging interface IPv6 address
Important notices
Before you add a new report, we ask you kindly to acknowledge the following:
- [X] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
- [X] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue
Describe the bug
When pinging the IPv6 address of any interface (WAN, LAN, Wireguard), the kernel panics and router reboot. I restored the same configuration file on 24.1 and no issue with 24.1.
To Reproduce
Steps to reproduce the behavior:
- Go to Overview
- Check the IPv6 address of the LAN interface
- From a computer on the LAN, ping the interface
- Router reboots
Expected behavior
Router shall respond to ping and not crash
Describe alternatives you considered
N/A
Screenshots
N/A
Relevant log files
Have been communicated through the crash reporter.
Additional context
I am using Intel I226 LAN (igc) with VLAN enabled.
Environment
OPNsense 24.7 Beta (amd64) Pinged from Windows 11 client.
ddb.txt06000014000014635311227 7076 ustarrootwheeldb:0:kdb.enter.default> run lockinfo
db:1:lockinfo> show locks
No such command; use "help" to list available commands
db:1:lockinfo> show alllocks
No such command; use "help" to list available commands
db:1:lockinfo> show lockedvnods
Locked vnodes
db:0:kdb.enter.default> show pcpu
cpuid = 3
dynamic pcpu = 0xfffffe00b9e6fc40
curthread = 0xfffff80001abe000: pid 0 tid 100018 critnest 1 "if_io_tqg_3"
curpcb = 0xfffff80001abe520
fpcurthread = none
idlethread = 0xfffff80001aef000: tid 100006 "idle: cpu3"
self = 0xffffffff82c13000
curpmap = 0xffffffff81b81670
tssp = 0xffffffff82c13384
rsp0 = 0xfffffe0038bd7000
kcr3 = 0x80000000313d1002
ucr3 = 0xffffffffffffffff
scr3 = 0x3efad7c4f
gs32p = 0xffffffff82c13404
ldt = 0xffffffff82c13444
tss = 0xffffffff82c13434
curvnet = 0xfffff800012a5980
db:0:kdb.enter.default> bt
Tracing pid 0 tid 100018 td 0xfffff80001abe000
kdb_enter() at kdb_enter+0x33/frame 0xfffffe0038bd6660
panic() at panic+0x43/frame 0xfffffe0038bd66c0
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe0038bd6720
trap_pfault() at trap_pfault+0x46/frame 0xfffffe0038bd6770
calltrap() at calltrap+0x8/frame 0xfffffe0038bd6770
--- trap 0xc, rip = 0xffffffff80ddb5d7, rsp = 0xfffffe0038bd6840, rbp = 0xfffffe0038bd6970 ---
ip6_forward() at ip6_forward+0x2a7/frame 0xfffffe0038bd6970
ip6_input() at ip6_input+0x11f/frame 0xfffffe0038bd6a50
netisr_dispatch_src() at netisr_dispatch_src+0x9e/frame 0xfffffe0038bd6aa0
ether_demux() at ether_demux+0x149/frame 0xfffffe0038bd6ad0
ether_nh_input() at ether_nh_input+0x36a/frame 0xfffffe0038bd6b30
netisr_dispatch_src() at netisr_dispatch_src+0x9e/frame 0xfffffe0038bd6b80
ether_input() at ether_input+0x56/frame 0xfffffe0038bd6bd0
ether_demux() at ether_demux+0x97/frame 0xfffffe0038bd6c00
ether_nh_input() at ether_nh_input+0x36a/frame 0xfffffe0038bd6c60
netisr_dispatch_src() at netisr_dispatch_src+0x9e/frame 0xfffffe0038bd6cb0
ether_input() at ether_input+0x56/frame 0xfffffe0038bd6d00
iflib_rxeof() at iflib_rxeof+0xc0e/frame 0xfffffe0038bd6e00
_task_fn_rx() at _task_fn_rx+0x72/frame 0xfffffe0038bd6e40
gtaskqueue_run_locked() at gtaskqueue_run_locked+0x14e/frame 0xfffffe0038bd6ec0
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc2/frame 0xfffffe0038bd6ef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe0038bd6f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0038bd6f30
--- trap 0x9d9d3e64, rip = 0xb414471ee4b2474b, rsp = 0x3113c21961b5c24c, rbp = 0x5647a54d06e1a518 ---
From a different user, same same but different:
ddb.txt06000014000014633270447 7104 ustarrootwheeldb:0:kdb.enter.default> run lockinfo
db:1:lockinfo> show locks
No such command; use "help" to list available commands
db:1:lockinfo> show alllocks
No such command; use "help" to list available commands
db:1:lockinfo> show lockedvnods
Locked vnodes
db:0:kdb.enter.default> show pcpu
cpuid = 3
dynamic pcpu = 0xfffffe00b6a73c40
curthread = 0xfffff80001a30000: pid 12 tid 100044 critnest 1 "swi1: netisr 3"
curpcb = 0xfffff80001a30520
fpcurthread = none
idlethread = 0xfffff80001a7d000: tid 100006 "idle: cpu3"
self = 0xffffffff82c13000
curpmap = 0xffffffff81b81670
tssp = 0xffffffff82c13384
rsp0 = 0xfffffe00dfe64000
kcr3 = 0xffffffffffffffff
ucr3 = 0xffffffffffffffff
scr3 = 0x0
gs32p = 0xffffffff82c13404
ldt = 0xffffffff82c13444
tss = 0xffffffff82c13434
curvnet = 0xfffff80001268980
db:0:kdb.enter.default> bt
Tracing pid 12 tid 100044 td 0xfffff80001a30000
kdb_enter() at kdb_enter+0x33/frame 0xfffffe00dfe63ac0
panic() at panic+0x43/frame 0xfffffe00dfe63b20
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe00dfe63b80
trap_pfault() at trap_pfault+0x46/frame 0xfffffe00dfe63bd0
calltrap() at calltrap+0x8/frame 0xfffffe00dfe63bd0
--- trap 0xc, rip = 0xffffffff80ddaee4, rsp = 0xfffffe00dfe63ca0, rbp = 0xfffffe00dfe63d10 ---
ip6_tryforward() at ip6_tryforward+0x264/frame 0xfffffe00dfe63d10
ip6_input() at ip6_input+0x537/frame 0xfffffe00dfe63df0
swi_net() at swi_net+0x138/frame 0xfffffe00dfe63e60
ithread_loop() at ithread_loop+0x257/frame 0xfffffe00dfe63ef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe00dfe63f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00dfe63f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
And the last one:
ddb.txt06000014000014632525416 7102 ustarrootwheeldb:0:kdb.enter.default> run lockinfo
db:1:lockinfo> show locks
No such command; use "help" to list available commands
db:1:lockinfo> show alllocks
No such command; use "help" to list available commands
db:1:lockinfo> show lockedvnods
Locked vnodes
db:0:kdb.enter.default> show pcpu
cpuid = 2
dynamic pcpu = 0xfffffe00b6a64c40
curthread = 0xfffff800b9f04000: pid 8092 tid 100997 critnest 1 "AdGuardHome"
curpcb = 0xfffff800b9f04520
fpcurthread = 0xfffff800b9f04000: pid 8092 "AdGuardHome"
idlethread = 0xfffff80001a7d740: tid 100005 "idle: cpu2"
self = 0xffffffff82c12000
curpmap = 0xfffff8001a8acd38
tssp = 0xffffffff82c12384
rsp0 = 0xfffffe0123531000
kcr3 = 0xffffffffffffffff
ucr3 = 0xffffffffffffffff
scr3 = 0x0
gs32p = 0xffffffff82c12404
ldt = 0xffffffff82c12444
tss = 0xffffffff82c12434
curvnet = 0xfffff80001268980
db:0:kdb.enter.default> bt
Tracing pid 8092 tid 100997 td 0xfffff800b9f04000
kdb_enter() at kdb_enter+0x33/frame 0xfffffe01235308d0
panic() at panic+0x43/frame 0xfffffe0123530930
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe0123530990
trap_pfault() at trap_pfault+0x46/frame 0xfffffe01235309e0
calltrap() at calltrap+0x8/frame 0xfffffe01235309e0
--- trap 0xc, rip = 0xffffffff80dd993a, rsp = 0xfffffe0123530ab0, rbp = 0xfffffe0123530be0 ---
in6_selectsrc() at in6_selectsrc+0x65a/frame 0xfffffe0123530be0
in6_selectsrc_socket() at in6_selectsrc_socket+0x41/frame 0xfffffe0123530c20
in6_pcbconnect() at in6_pcbconnect+0x172/frame 0xfffffe0123530ca0
udp6_connect() at udp6_connect+0x2d4/frame 0xfffffe0123530d20
soconnectat() at soconnectat+0xb1/frame 0xfffffe0123530d60
kern_connectat() at kern_connectat+0xe3/frame 0xfffffe0123530dc0
sys_connect() at sys_connect+0x81/frame 0xfffffe0123530e00
amd64_syscall() at amd64_syscall+0x100/frame 0xfffffe0123530f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0123530f30
--- syscall (98, FreeBSD ELF64, connect), rip = 0x4906bf, rsp = 0x86c130d58, rbp = 0x86c130d58 ---
@fichtner not sure if it's the same thing, but the ip6_forward() looks suspicious when this traffic was intended for the host itself. maybe @belotv would be so kind to add some context to the ticket, like features that are being used and if any ipv6 forwarding or NAT are in play here. A (censored) ifconfig is also helpful. Just adding an ipv6 address to an interface and pinging it isn't enough to reproduce this (at least not from my end).
I collected all submitted panics for the scope of this ticket. One of this might be related to https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=279653 https://reviews.freebsd.org/D45690
Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 03
fault virtual address = 0x10
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80de6a06
stack pointer = 0x28:0xfffffe00629a7b80
frame pointer = 0x28:0xfffffe00629a7bb0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 2 (clock (0))
rdi: fffff8008978c800 rsi: 000000000000001c rdx: fffff80014e7d278
rcx: fffff8008978c800 r8: 00000000ffffff5e r9: 0000000000000018
rax: 0000000000000000 rbx: 0000000000000000 rbp: fffffe00629a7bb0
r10: fffff8003a72abe0 r11: fffffe00a731a000 r12: 000000000000ff70
r13: fffff8000165e740 r14: fffffe00629a7b8c r15: fffff8001649b284
trap number = 12
panic: page fault
cpuid = 3
time = 1738754035
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00629a7870
vpanic() at vpanic+0x131/frame 0xfffffe00629a79a0
panic() at panic+0x43/frame 0xfffffe00629a7a00
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe00629a7a60
trap_pfault() at trap_pfault+0x46/frame 0xfffffe00629a7ab0
calltrap() at calltrap+0x8/frame 0xfffffe00629a7ab0
--- trap 0xc, rip = 0xffffffff80de6a06, rsp = 0xfffffe00629a7b80, rbp = 0xfffffe00629a7bb0 ---
in6_selecthlim() at in6_selecthlim+0x96/frame 0xfffffe00629a7bb0
tcp_default_output() at tcp_default_output+0x1d1b/frame 0xfffffe00629a7d70
tcp_timer_rexmt() at tcp_timer_rexmt+0x554/frame 0xfffffe00629a7dd0
tcp_timer_enter() at tcp_timer_enter+0x101/frame 0xfffffe00629a7e10
softclock_call_cc() at softclock_call_cc+0x12c/frame 0xfffffe00629a7ec0
softclock_thread() at softclock_thread+0xe5/frame 0xfffffe00629a7ef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe00629a7f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00629a7f30
--- trap 0x269f0f60, rip = 0x8269f2250, rsp = 0x8269a5bb0, rbp = 0x8269eec70 ---
KDB: enter: panic
I am travelling today but will provide the ifconfig tomorrow. I reverted back to 24.1 for the moment. I am using wireguard tunnel with ipv4+ipv6, so indeed I have outbound nat. I am using the configuration from https://docs.opnsense.org/manual/how-tos/wireguard-selective-routing.html and exposing the wireguard connection on VLAN 2. (I tried with NPTv6 too and have the same issue on 24.7 but loading the very same config file on 24.1 works fine). FYI, my wireguard endpoint is an IPv6.
I also have traffic shaper enabled (queues/pipes).
First two: ip6_input() has no significant changes and since both panics lead to ip6_forward() AND ip6_tryforward() it's suspicious as both called functions can't really crash for the same reason in separate code paths unless there was a problem in the path of ip6_input() before from stock FreeBSD. The commit f257b8d7 isn't on stable/14 either which makes it a prime candidate and will be included in RC2.
Third one: caused by adguardhome, which we don't provide and no idea what FreeBSD version it was built against (not 14.1 judging by the timing). No changes in FreeBSD in that area either so wait and see.
The ip6_input() one is still happening. I'll issue a debug kernel in RC2 to get to the bottom of it.
ip6_forward() commit 9cb6d71f6a41d but still unclear about the ip6_tryforward()
ip6_tryforward() seems to be buggy upstream, because https://redmine.pfsense.org/issues/15640
Not sure if I should use this issue or create a new one, but it looks related, so I'll start here. TIL (actually yesterday) that if I attempt to "Reload" my WAN (pppoe) interface from the OPNsense (25.1.2) WebUI's Interfaces Overview, it crashes in ip6_tryforward(). It seems to be consistent - or at least has happened 3 out of 3 attempts...
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address = 0x10
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80de7b70
stack pointer = 0x28:0xfffffe0003791a30
frame pointer = 0x28:0xfffffe0003791aa0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 0 (if_io_tqg_1)
rdi: fffff80027190000 rsi: 000000000000001c rdx: fffff8006dc98a78
rcx: 0000000000000280 r8: 00000000ffffffb0 r9: 0000000000000018
rax: 0000000000000000 rbx: fffff8004739866e rbp: fffffe0003791aa0
r10: fffff8002919e960 r11: fffffe007b0b8000 r12: fffff80047398686
r13: fffff800034ef000 r14: fffffe0003791a40 r15: fffff8000391c800
trap number = 12
panic: page fault
cpuid = 1
time = 1740847093
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0003791720
vpanic() at vpanic+0x131/frame 0xfffffe0003791850
panic() at panic+0x43/frame 0xfffffe00037918b0
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe0003791910
trap_pfault() at trap_pfault+0x46/frame 0xfffffe0003791960
calltrap() at calltrap+0x8/frame 0xfffffe0003791960
--- trap 0xc, rip = 0xffffffff80de7b70, rsp = 0xfffffe0003791a30, rbp = 0xfffffe0003791aa0 ---
ip6_tryforward() at ip6_tryforward+0x280/frame 0xfffffe0003791aa0
ip6_input() at ip6_input+0x684/frame 0xfffffe0003791b80
netisr_dispatch_src() at netisr_dispatch_src+0x9e/frame 0xfffffe0003791bd0
ether_demux() at ether_demux+0x149/frame 0xfffffe0003791c00
ether_nh_input() at ether_nh_input+0x36a/frame 0xfffffe0003791c60
netisr_dispatch_src() at netisr_dispatch_src+0x9e/frame 0xfffffe0003791cb0
ether_input() at ether_input+0x56/frame 0xfffffe0003791d00
iflib_rxeof() at iflib_rxeof+0xc1e/frame 0xfffffe0003791e00
_task_fn_rx() at _task_fn_rx+0x72/frame 0xfffffe0003791e40
gtaskqueue_run_locked() at gtaskqueue_run_locked+0x14e/frame 0xfffffe0003791ec0
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc2/frame 0xfffffe0003791ef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe0003791f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0003791f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
@dseven would you mind installing a debug kernel and sending me the resulting /var/crash/vmcore.* files? It should be easy to fix with a working vmcore, but I don’t have one so far. Also still suspecting a FreeBSD issue. PPPoE maybe?
# opnsense-update -zkr dbg-25.1.2
Thanks, Franco
Of course I can't get it to reproduce with the debug kernel! Urgh! I'll keep trying.
PPPoE and IPv6 seem possible factors. Just a complete guess, but maybe it has something to do with the pppoe device completely disappearing / getting recreated on "reload", where others (physical NICs) normally would persist.
Finally got it to happen with the debug kernel. Emailed vmcore to you (@fichtner) LMK if you didn't get it...
@dseven perfect, got it and it works with the right panic path (sometimes these vmcores segfault in python). I will look at it tomorrow morning.
Cheers, Franco
Just preliminary info since was curious. This line:
https://github.com/opnsense/src/blob/b8ab1d06e8048a68c09f4e182868d2d5f06f6263/sys/netinet6/ip6_fastfwd.c#L194
(kgdb) p nifp->if_xname
$9 = "pppoe0\000\000\000\000\000\000\000\000\000"
Through if_getifdata() magic inside the convoluted macro IN6_LINKMTU()
(kgdb) p nifp->if_afdata
$16 = {0x0 <repeats 44 times>}
So bad pointer, half attached pppoe0
@dseven I went through a few iterations of a patch arriving at 5dc500b. I'm doubting what FreeBSD would do so I'm raising a bug report and review there shortly.
However, I've tested the code in my production environment and I'm writing this from the kernel booted so if you can test this kernel to see if the error is going away that would be fantastic!
# opnsense-update -zkr 25.1.2-donottest
(reboot)
It's called do-not-test just because I know some like to test snapshot kernels so I wanted to make clear this is not for blackbox testing. ;)
Cheers, Franco
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285129 https://reviews.freebsd.org/D49212
Looks good so far! I've "Reloaded" a handful of times. I'll keep this running and look out for side-effects, but nothing obvious so far.
@dseven thanks a ton! ❤ might take a while to be sure these are gone completely, but definitely promising
Cheers, Franco
I think we can close this defect, I kind of forgot its existence, sorry for that. The goal was not to catch all kernel panics events as it would be really difficult to keep track of bug resolution. The initial panic I reported was indeed linked to ip6_forward() which got fixed in commit 9cb6d71. I will let @fichtner close it once you are done with the second bug reported by @dseven.
@dseven how's that kernel holding up for you? FreeBSD wanted to do something different, maybe you can assist with testing next week. For reference: https://reviews.freebsd.org/D49359
@fichtner I wasn't able to reproduce the issue with your test kernel, then I kinda forgot about it and upgraded to 25.1.3 when that was released😉 I would be willing to test alternative solutions if you can provide kernel packages👍
@dseven ok, great, can you try this one? https://reviews.freebsd.org/D49359
# opnsense-update -zkr 25.1.3-teardown
Cheers, Franco
@fichtner I was able to reproduce the crash with this one - looks the same as before (I think?)
db:0:kdb.enter.default> bt
Tracing pid 0 tid 100012 td 0xfffff80003507000
kdb_enter() at kdb_enter+0x33/frame 0xfffffe0003791850
panic() at panic+0x43/frame 0xfffffe00037918b0
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe0003791910
trap_pfault() at trap_pfault+0x46/frame 0xfffffe0003791960
calltrap() at calltrap+0x8/frame 0xfffffe0003791960
--- trap 0xc, rip = 0xffffffff80de7bf0, rsp = 0xfffffe0003791a30, rbp = 0xfffffe0003791aa0 ---
ip6_tryforward() at ip6_tryforward+0x280/frame 0xfffffe0003791aa0
ip6_input() at ip6_input+0x684/frame 0xfffffe0003791b80
netisr_dispatch_src() at netisr_dispatch_src+0x9e/frame 0xfffffe0003791bd0
ether_demux() at ether_demux+0x149/frame 0xfffffe0003791c00
ether_nh_input() at ether_nh_input+0x36a/frame 0xfffffe0003791c60
netisr_dispatch_src() at netisr_dispatch_src+0x9e/frame 0xfffffe0003791cb0
ether_input() at ether_input+0x56/frame 0xfffffe0003791d00
iflib_rxeof() at iflib_rxeof+0xc1e/frame 0xfffffe0003791e00
_task_fn_rx() at _task_fn_rx+0x72/frame 0xfffffe0003791e40
gtaskqueue_run_locked() at gtaskqueue_run_locked+0x14e/frame 0xfffffe0003791ec0
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc2/frame 0xfffffe0003791ef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe0003791f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0003791f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
@dseven thanks, reported upstream. the "25.1.2-donottest" is still there if you want to keep using it. I'll rebase this on 25.1.3 later.
I'm seeing this panic as well, running OPNsense 25.1.5_5-amd64. It's not consistent though. I was able to trigger it by saving a PPPoE interface, but the immediately after the reboot a save did not trigger the panic again.
I also haven't been able trigger it by pinging the LAN interface.
Anything to do beyond waiting for reviews upstream?
The POC works, but is deemed too problematic for general inclusion. I could rewrite the particular spot to stop failing instead because I think we won't see a fix before FreeBSD 15.
@deviantintegral @dseven how about this then https://github.com/opnsense/src/commit/5afe2ee0908
kernel in a bit, need to test for sanity first