src icon indicating copy to clipboard operation
src copied to clipboard
verified publisher icon opnsense

Page Fault Crashes after upgrade to 23.7.2

Open kb8080 opened this issue 2 years ago • 6 comments

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

  • [X] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
  • [X] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

I recently upgraded this weekend to 23.7.2 and started experiencing crashes which appear to be caused by page faults. Nothing configuration-wise was changed while on 23.7.1 prior to the upgrade. Now one thing of note is that I was on backup portable generator power due to the severe storms that rolled through the midwest late last week and into the weekend. The only reason I mention this is that I reverted the kernel/os back to 23.7.1 a couple of hours before grid power came back and the firewall has been stable since. I still need to look into whether I had some crashes after the reversion or if this was the solution or something else. I don't know if this is a coincidence or if there was some power sag that the device was super sensitive to? Nothing else in the house experience any issues including the network equipment that was plugged into the same power strip as the firewall. Also, I was running on 23.7.1 while on generator for about 12hrs stable prior to the upgrade to 23.7.2 event.

I'm hoping that someone may be able to decipher the provided dump information and some possible causes for the crashing.

Tip: to validate your setup was working with the previous version, use opnsense-revert (https://docs.opnsense.org/manual/opnsense_tools.html#opnsense-revert)

To Reproduce

Upgraded to 23.7.2

Expected behavior

Stable firewall operation.

Describe alternatives you considered

Considered re-flashing firewall to 23.7 and restoring from backup config.

Screenshots

N/A

Relevant log files

Had to truncate these slightly so it would fit within the comment char limit so if I redacted something critical I can post separately.

/var/crash/info.35: Dump header from device: /dev/ada0s1b Architecture: amd64 Architecture Version: 4 Dump Length: 80896 Blocksize: 512 Compression: none Dumptime: 2023-08-27 06:07:31 -0400 Hostname: gw1.example.com Magic: FreeBSD Text Dump Version String: FreeBSD 13.2-RELEASE-p2 stable/23.7-n254761-4b4f06e3731 SMP Panic String: page fault Dump Parity: 827694708 Bounds: 35 Dump Status: good

/var/crash/textdump.tar.35: ddb.txt06000014000014472620143 7075 ustarrootwheeldb:0:kdb.enter.default> run lockinfo db:1:lockinfo> show locks No such command; use "help" to list available commands db:1:lockinfo> show alllocks No such command; use "help" to list available commands db:1:lockinfo> show lockedvnods Locked vnodes db:0:kdb.enter.default> show pcpu cpuid = 0 dynamic pcpu = 0x9649c0 curthread = 0xfffffe0083862e40: pid 12 tid 100042 critnest 1 "swi1: netisr 2" curpcb = 0xfffffe0083863350 fpcurthread = none idlethread = 0xfffffe00105cd3a0: tid 100003 "idle: cpu0" self = 0xffffffff82610000 curpmap = 0xffffffff81ea6bb8 tssp = 0xffffffff82610384 rsp0 = 0xfffffe008373c000 kcr3 = 0x2519000 ucr3 = 0xffffffffffffffff scr3 = 0xa9a53000 gs32p = 0xffffffff82610404 ldt = 0xffffffff82610444 tss = 0xffffffff82610434 curvnet = 0xfffff800031bbe00 db:0:kdb.enter.default> bt Tracing pid 12 tid 100042 td 0xfffffe0083862e40 kdb_enter() at kdb_enter+0x37/frame 0xfffffe008373b8f0 vpanic() at vpanic+0x182/frame 0xfffffe008373b940 panic() at panic+0x43/frame 0xfffffe008373b9a0 trap_fatal() at trap_fatal+0x387/frame 0xfffffe008373ba00 trap_pfault() at trap_pfault+0x4f/frame 0xfffffe008373ba60 calltrap() at calltrap+0x8/frame 0xfffffe008373ba60 --- trap 0xc, rip = 0xffffffff823896cf, rsp = 0xfffffe008373bb30, rbp = 0xfffffe008373bb90 --- pf_test_state_udp() at pf_test_state_udp+0x28f/frame 0xfffffe008373bb90 pf_test() at pf_test+0xc57/frame 0xfffffe008373bd00 pf_check_in() at pf_check_in+0x25/frame 0xfffffe008373bd20 pfil_run_hooks() at pfil_run_hooks+0x97/frame 0xfffffe008373bd60 ip_input() at ip_input+0x799/frame 0xfffffe008373bdf0 swi_net() at swi_net+0x12b/frame 0xfffffe008373be60 ithread_loop() at ithread_loop+0x25a/frame 0xfffffe008373bef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe008373bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008373bf30 --- trap 0x3249811e, rip = 0x108ebb17154ccb14, rsp = 0x1fe99f2508cc494, rbp = 0x468d176d4d40ff0d --- db:0:kdb.enter.default> ps pid ppid pgrp uid state wmesg wchan cmd 56698 90983 90029 0 S nanslp 0xffffffff81ec60e2 sleep 3520 14970 3520 0 S select 0xfffff8001118fd40 python3.9 3334 14970 3334 0 S select 0xfffff80162fd51c0 python3.9 92310 83446 83446 0 S (threaded) python3.9 100419 S accept 0xfffff80004f248b8 python3.9 83446 1 83446 0 Ss wait 0xfffff800a9b22aa0 python3.9 90983 90029 90029 0 S wait 0xfffff800a92e2aa0 sh 90029 1 90029 0 Ss piperd 0xfffff80011f6f000 daemon 84282 79622 79622 0 S accept 0xfffff80004ce1158 php-cgi 83309 79622 79622 0 S accept 0xfffff80004ce1158 php-cgi 82497 79372 79372 0 S accept 0xfffff80004f24c68 php-cgi 81872 79622 79622 0 S accept 0xfffff80004ce1158 php-cgi 81652 79372 79372 0 S accept 0xfffff80004f24c68 php-cgi 80654 79372 79372 0 S accept 0xfffff80004f24c68 php-cgi 79622 78865 79622 0 Ss wait 0xfffff80004afa550 php-cgi 79372 78865 79372 0 Ss wait 0xfffff80011f2e000 php-cgi 78865 1 78545 0 S kqread 0xfffff800116be600 lighttpd 74704 99403 74704 0 Ss+ ttyin 0xfffff8000405ccb0 sh 99403 55631 99403 0 Ss select 0xfffff80011cbd6c0 sshd 52347 1 52347 0 Ss+ ttyin 0xfffff800036e9cb0 getty 51752 1 51752 0 Ss+ ttyin 0xfffff800036e84b0 getty 51061 1 51061 0 Ss+ ttyin 0xfffff800036e8cb0 getty 50390 1 50390 0 Ss+ ttyin 0xfffff800036e74b0 getty 49982 1 49982 0 Ss+ ttyin 0xfffff800036e7cb0 getty 49199 1 49199 0 Ss+ ttyin 0xfffff8000372f4b0 getty 48813 1 48813 0 Ss+ ttyin 0xfffff8000372fcb0 getty 47781 1 47781 0 Ss+ ttyin 0xfffff8000372e4b0 getty 15558 1 15558 0 Ss bpf 0xfffff80094fdb200 filterlog 15519 1 15519 0 Ss nanslp 0xffffffff81ec60e0 cron 6174 1 6174 0 Ss (threaded) dpinger 100173 S uwait 0xfffff80004046480 dpinger 100365 S sbwait 0xfffff80004cd9924 dpinger 100366 S nanslp 0xffffffff81ec60e2 dpinger 100367 S nanslp 0xffffffff81ec60e2 dpinger 100368 S accept 0xfffff80004e7d8b8 dpinger 12958 12507 12507 0 S (threaded) crowdsec-firewall-b 100238 S uwait 0xfffff80011c0df00 crowdsec-firewall-b 100333 S uwait 0xfffff80095dd4d80 crowdsec-firewall-b 100334 S uwait 0xfffff80067736700 crowdsec-firewall-b 100335 S kqread 0xfffff800115dba00 crowdsec-firewall-b 100336 S uwait 0xfffff80004046280 crowdsec-firewall-b 100338 S uwait 0xfffff80004dddd00 crowdsec-firewall-b 100339 S uwait 0xfffff80095ed3c80 crowdsec-firewall-b 100340 S uwait 0xfffff80095d67180 crowdsec-firewall-b 100341 S uwait 0xfffff80095d67080 crowdsec-firewall-b 100342 S uwait 0xfffff8007ece6f00 crowdsec-firewall-b 12507 1 12507 0 Ss piperd 0xfffff80004a7eba0 daemon 88916 88807 88807 884 S (threaded) telegraf 100313 S uwait 0xfffff80067113400 telegraf 100328 S uwait 0xfffff80095fb5500 telegraf 100329 S uwait 0xfffff80004ef9300 telegraf 100330 S uwait 0xfffff80004ab8800 telegraf 100331 S uwait 0xfffff80095dd4900 telegraf 100332 S uwait 0xfffff80004dddf00 telegraf 100337 S uwait 0xfffff80095ed3780 telegraf 100344 S uwait 0xfffff80095fb5300 telegraf 100353 S uwait 0xfffff80004878f00 telegraf 100363 S uwait 0xfffff800671e6680 telegraf 100379 S kqread 0xfffff80011ae5300 telegraf 88807 1 88807 0 Ss piperd 0xfffff80004b00ba0 daemon 65444 1 64981 0 S kqread 0xfffff80011c1ad00 lighttpd 42389 41968 41968 0 S (threaded) crowdsec 100249 S kqread 0xfffff800115d2600 crowdsec 100318 S uwait 0xfffff80004047c00 crowdsec 100319 S kqread 0xfffff800115d2300 crowdsec 100320 S kqread 0xfffff800115d2400 crowdsec 100321 S uwait 0xfffff80095fb6280 crowdsec 100322 S uwait 0xfffff800955a0080 crowdsec 100323 S uwait 0xfffff80095dd4b80 crowdsec 100324 S uwait 0xfffff80095dd4a80 crowdsec 100325 S kqread 0xfffff800115d2500 crowdsec 100371 S kqread 0xfffff80011aea000 crowdsec 100372 S uwait 0xfffff800671e6e00 crowdsec 100373 S uwait 0xfffff800678fe280 crowdsec 100374 S uwait 0xfffff80004878200 crowdsec 100403 S uwait 0xfffff800678fe900 crowdsec 41968 1 41968 0 Ss piperd 0xfffff80003bfaba0 daemon 36724 1 36724 59 Ss (threaded) unbound 100157 S kqread 0xfffff800115da300 unbound 100315 S kqread 0xfffff80011c1ae00 unbound 100316 S kqread 0xfffff80011ae5500 unbound 100317 S kqread 0xfffff80011aea300 unbound 52310 1 52310 0 Ss (threaded) ntpd 100285 S select 0xfffff80011190e40 ntpd 11931 10581 11931 542 Ss select 0xfffff8007e799cc0 flowd 10581 1 10581 0 Ss sbwait 0xfffff80004e3ecd4 flowd 7420 1 7420 0 Ss select 0xfffff80095d67440 python3.9 83799 83648 83648 65534 S sbwait 0xfffff80004e17574 samplicate 83648 1 83648 0 Ss piperd 0xfffff80004b04000 daemon 68437 0 0 0 DL (threaded) [ng_queue] 100105 D sleep 0xffffffff8273a850 [ng_queue0] 100256 D sleep 0xffffffff8273a850 [ng_queue1] 100257 D sleep 0xffffffff8273a850 [ng_queue2] 100258 D sleep 0xffffffff8273a850 [ng_queue3] 64897 1 18 0 S+ sbwait 0xfffff80004f251c4 udpbroadcastrelay 33897 1 33897 0 Ss select 0xfffff80011cbdcc0 openvpn 29568 1 29568 0 Ss select 0xfffff8007e5bc3c0 openvpn 27405 26542 26542 0 S (threaded) charon 100192 S sigwait 0xfffff80004b880b0 charon 100210 S uwait 0xfffff80011190880 charon 100211 S select 0xfffff80095fb5240 charon 100212 S select 0xfffff80095dd36c0 charon 100213 S uwait 0xfffff80095dd5800 charon 100214 S uwait 0xfffff80095dd5780 charon 100215 S uwait 0xfffff80095dd3200 charon 100216 S uwait 0xfffff80095e6dd80 charon 100217 S uwait 0xfffff80004048800 charon 100218 S uwait 0xfffff80095dd4400 charon 100219 S uwait 0xfffff80004048900 charon 100220 S uwait 0xfffff80095dd4280 charon 100221 S uwait 0xfffff80095dd4980 charon 100222 S uwait 0xfffff80095dd4080 charon 100223 S uwait 0xfffff80095dd5e80 charon 100224 S uwait 0xfffff80095dd5b80 charon 100225 S uwait 0xfffff80095fb5c00 charon 26542 1 26542 0 Ss piperd 0xfffff80004ba35d0 daemon 1410 1384 1384 0 S nanslp 0xffffffff81ec60e3 php 1384 1 1384 0 Ss piperd 0xfffff800110a08b8 daemon 69906 1 69906 136 Ss select 0xfffff80004ea1040 dhcpd 55631 1 55631 0 Ss select 0xfffff80004ffd840 sshd 14970 14494 14970 0 Ss (threaded) syslog-ng 100145 S kqread 0xfffff800047d7100 syslog-ng 100444 S kqread 0xfffff80011ae6d00 syslog-ng 100461 S kqread 0xfffff80162b4bc00 syslog-ng 14494 1 14130 0 S wait 0xfffff80067140550 syslog-ng 551 1 551 0 Ss select 0xfffff80011c0d2c0 devd 17 0 0 0 DL vlruwt 0xfffff800036d0aa0 [vnlru] 16 0 0 0 DL syncer 0xffffffff81f16450 [syncer] 9 0 0 0 DL (threaded) [bufdaemon] 100083 D psleep 0xffffffff81f15508 [bufdaemon] 100086 D - 0xffffffff81a0aec0 [bufspacedaemon-0] 100087 D - 0xffffffff81a13180 [bufspacedaemon-1] 100088 D - 0xffffffff81a1b440 [bufspacedaemon-2] 100089 D - 0xffffffff81a23700 [bufspacedaemon-3] 100104 D sdflush 0xfffff8000405a0e8 [/ worker] 8 0 0 0 DL psleep 0xffffffff81f3f390 [vmdaemon] 7 0 0 0 DL (threaded) [pagedaemon] 100081 D psleep 0xffffffff81f337f8 [dom0] 100084 D launds 0xffffffff81f33804 [laundry: dom0] 100085 D umarcl 0xffffffff80febc80 [uma] 6 0 0 0 DL - 0xffffffff81cfd300 [rand_harvestq] 5 0 0 0 DL pftm 0xffffffff8237f900 [pf purge] 15 0 0 0 DL (threaded) [usb] 100066 D - 0xfffffe008e966cf0 [usbus0] 100067 D - 0xfffffe008e966d48 [usbus0] 100068 D - 0xfffffe008e966da0 [usbus0] 100069 D - 0xfffffe008e966df8 [usbus0] 100070 D - 0xfffffe008e966e50 [usbus0] 4 0 0 0 DL (threaded) [cam] 100063 D - 0xffffffff81bd43c0 [doneq0] 100080 D - 0xffffffff81bd4290 [scanner] 14 0 0 0 DL seqstat 0xfffff80003992c88 [sequencer 00] 3 0 0 0 DL (threaded) [crypto] 100057 D crypto_ 0xffffffff81f30cf0 [crypto] 100058 D crypto_ 0xfffff800031b1330 [crypto returns 0] 100059 D crypto_ 0xfffff800031b1390 [crypto returns 1] 100060 D crypto_ 0xfffff800031b13f0 [crypto returns 2] 100061 D crypto_ 0xfffff800031b1450 [crypto returns 3] 13 0 0 0 DL (threaded) [geom] 100050 D - 0xffffffff81ea5858 [g_event] 100051 D - 0xffffffff81ea5860 [g_up] 100052 D - 0xffffffff81ea5868 [g_down] 2 0 0 0 DL (threaded) [KTLS] 100036 D - 0xfffff800031b0800 [thr_0] 100037 D - 0xfffff800031b0880 [thr_1] 100038 D - 0xfffff800031b0900 [thr_2] 100039 D - 0xfffff800031b0980 [thr_3] 12 0 0 0 RL (threaded) [intr] 100018 I [swi6: task queue] 100020 I [swi6: Giant taskq] 100022 I [swi5: fast taskq] 100040 I [swi1: netisr 0] 100041 I [swi1: netisr 1] 100042 Run CPU 0 [swi1: netisr 2] 100043 I [swi1: netisr 3] 100044 I [swi4: clock (0)] 100045 I [swi4: clock (1)] 100046 I [swi4: clock (2)] 100047 I [swi4: clock (3)] 100048 I [swi3: busdma] 100064 I [irq87: ahci0] 100065 I [irq23: ehci0] 100072 I [swi1: pf send] 100073 I [swi1: pfsync] 100136 I [irq18: ichsmb0] 11 0 0 0 RL (threaded) [idle] 100003 CanRun [idle: cpu0] 100004 Run CPU 1 [idle: cpu1] 100005 Run CPU 2 [idle: cpu2] 100006 Run CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xfffff800036d0000 [init] 10 0 0 0 DL audit_w 0xffffffff81f31210 [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0xffffffff81ea5e08 [swapper] 100007 D - 0xfffff800036ba800 [softirq_0] 100008 D - 0xfffff800036ba600 [softirq_1] 100009 D - 0xfffff800036ba400 [softirq_2] 100010 D - 0xfffff800036ba200 [softirq_3] 100011 D - 0xfffff800036ba000 [if_io_tqg_0] 100012 D - 0xfffff800031b8d00 [if_io_tqg_1] 100013 D - 0xfffff800031b8b00 [if_io_tqg_2] 100014 D - 0xfffff800031b8900 [if_io_tqg_3] 100015 D - 0xfffff800031b8700 [if_config_tqg_0] 100016 D - 0xfffff800031b8500 [aiod_kick taskq] 100017 D - 0xfffff800031b8300 [pci_hp taskq] 100019 D - 0xfffff800036ad200 [linuxkpi_irq_wq] 100021 D - 0xfffff800036aec00 [thread taskq] 100023 D - 0xfffff800036ae700 [inm_free taskq] 100024 D - 0xfffff800036ae500 [kqueue_ctx taskq] 100025 D - 0xfffff800036ae300 [in6m_free taskq] 100026 D - 0xfffff800036ae100 [linuxkpi_short_wq_0] 100027 D - 0xfffff800036ae100 [linuxkpi_short_wq_1] 100028 D - 0xfffff800036ae100 [linuxkpi_short_wq_2] 100029 D - 0xfffff800036ae100 [linuxkpi_short_wq_3] 100030 D - 0xfffff800036ae100 [linuxkpi_short_wq_4] 100031 D - 0xfffff800036afa00 [linuxkpi_long_wq_0] 100032 D - 0xfffff800036afa00 [linuxkpi_long_wq_1] 100033 D - 0xfffff800036afa00 [linuxkpi_long_wq_2] 100034 D - 0xfffff800036afa00 [linuxkpi_long_wq_3] 100035 D - 0xfffff800036afa00 [linuxkpi_long_wq_4] 100049 D - 0xfffff800036b7a00 [firmware taskq] 100053 D - 0xfffff800038cd900 [crypto_0] 100054 D - 0xfffff800038cd900 [crypto_1] 100055 D - 0xfffff800038cd900 [crypto_2] 100056 D - 0xfffff800038cd900 [crypto_3] 100074 D - 0xfffff800031b7b00 [mca taskq] 100075 D - 0xfffff80003a84000 [acpi_task_0] 100076 D - 0xfffff80003a84000 [acpi_task_1] 100077 D - 0xfffff80003a84000 [acpi_task_2] 100079 D - 0xfffff800038ce000 [CAM taskq] 100092 D - 0xfffff800047d7700 [/ trim] db:0:kdb.enter.default> alltrace

Tracing command kernel pid 0 tid 100000 td 0xffffffff81ea6360 sched_switch() at sched_switch+0x818/frame 0xffffffff82509ec0 mi_switch() at mi_switch+0xc2/frame 0xffffffff82509ee0 sleepq_timedwait() at sleepq_timedwait+0x2f/frame 0xffffffff82509f20 _sleep() at _sleep+0x1ce/frame 0xffffffff82509fa0 swapper() at swapper+0x69/frame 0xffffffff82509ff0 btext() at btext+0x22

Tracing command kernel pid 0 tid 100007 td 0xfffffe00105cb720 sched_switch() at sched_switch+0x818/frame 0xfffffe000ed92e50 mi_switch() at mi_switch+0xc2/frame 0xfffffe000ed92e70 msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe000ed92ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf4/frame 0xfffffe000ed92ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe000ed92f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000ed92f30 --- trap 0xa4502d2c, rip = 0xa808ca62f7579b22, rsp = 0x6e244610efe4b34c, rbp = 0x2190d989d137c7e3 ---

Tracing command kernel pid 0 tid 100008 td 0xfffffe00105cb000 sched_switch() at sched_switch+0x818/frame 0xfffffe000ed8de50 mi_switch() at mi_switch+0xc2/frame 0xfffffe000ed8de70 msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe000ed8dec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf4/frame 0xfffffe000ed8def0 fork_exit() at fork_exit+0x7e/frame 0xfffffe000ed8df30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000ed8df30 --- trap 0x21de2634, rip = 0x102645c0e813b5aa, rsp = 0x7a2b91db4025e5c6, rbp = 0x25902faf3ecd0a16 ---

Tracing command kernel pid 0 tid 100009 td 0xfffffe00105f9c80 sched_switch() at sched_switch+0x818/frame 0xfffffe000edd8e50 mi_switch() at mi_switch+0xc2/frame 0xfffffe000edd8e70 msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe000edd8ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf4/frame 0xfffffe000edd8ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe000edd8f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000edd8f30 --- trap 0x2a2e31d0, rip = 0x9546b0d7026d6156, rsp = 0x774374c48a7a7179, rbp = 0xf3207c5c9bf45e24 ---

Tracing command kernel pid 0 tid 100010 td 0xfffffe00105f9560 sched_switch() at sched_switch+0x818/frame 0xfffffe000edd3e50 mi_switch() at mi_switch+0xc2/frame 0xfffffe000edd3e70 msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe000edd3ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf4/frame 0xfffffe000edd3ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe000edd3f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000edd3f30 --- trap 0xb4600d4c, rip = 0x8eba87076d9dbeb6, rsp = 0xbbefedf8b8c3161b, rbp = 0x68194a57f7128b89 ---

Tracing command kernel pid 0 tid 100011 td 0xfffffe00105f8e40 sched_switch() at sched_switch+0x818/frame 0xfffffe000edcee50 mi_switch() at mi_switch+0xc2/frame 0xfffffe000edcee70 msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe000edceec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf4/frame 0xfffffe000edceef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe000edcef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000edcef30 --- trap 0x4838f5fd, rip = 0x575ecb32f5f78e63, rsp = 0xea0eb19040b264e9, rbp = 0xbd371f6b5d404b27 ---

Tracing command kernel pid 0 tid 100012 td 0xfffffe00105f8720 sched_switch() at sched_switch+0x818/frame 0xfffffe000edc9e50 mi_switch() at mi_switch+0xc2/frame 0xfffffe000edc9e70 msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe000edc9ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf4/frame 0xfffffe000edc9ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe000edc9f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000edc9f30 --- trap 0x21de2634, rip = 0x102645c0e813b5aa, rsp = 0x7a2b91cb4025e5c4, rbp = 0x25912eaf3fcd2e16 ---

Tracing command kernel pid 0 tid 100013 td 0xfffffe00105f8000 sched_switch() at sched_switch+0x818/frame 0xfffffe000edc4e50 mi_switch() at mi_switch+0xc2/frame 0xfffffe000edc4e70 msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe000edc4ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf4/frame 0xfffffe000edc4ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe000edc4f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000edc4f30 --- trap 0xd676c961, rip = 0x4ca3fcca9a075183, rsp = 0x268028c8320341ab, rbp = 0x2608291832a336da ---

Tracing command kernel pid 0 tid 100014 td 0xfffffe00105cf740 sched_switch() at sched_switch+0x818/frame 0xfffffe000edbfe50 mi_switch() at mi_switch+0xc2/frame 0xfffffe000edbfe70 msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe000edbfec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf4/frame 0xfffffe000edbfef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe000edbff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000edbff30 --- trap 0x2a2e3150, rip = 0x9546b0df026d6156, rsp = 0x774374ff8afa7079, rbp = 0xf3207c7f9bf45264 ---

Tracing command kernel pid 0 tid 100015 td 0xfffffe00105cf020 sched_switch() at sched_switch+0x818/frame 0xfffffe000edbae50 mi_switch() at mi_switch+0xc2/frame 0xfffffe000edbae70 msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe000edbaec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf4/frame 0xfffffe000edbaef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe000edbaf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000edbaf30 --- trap 0x3a1e11b0, rip = 0xb3f4fdba98a754c2, rsp = 0xa288dfeddd55df26, rbp = 0xbaa9efbebdd01e2e ---

Tracing command kernel pid 0 tid 100016 td 0xfffffe00105ce900 sched_switch() at sched_switch+0x818/frame 0xfffffe000edb5e20 mi_switch() at mi_switch+0xc2/frame 0xfffffe000edb5e40 _sleep() at _sleep+0x1ee/frame 0xfffffe000edb5ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe000edb5ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe000edb5f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000edb5f30 --- trap 0xb4600d4c, rip = 0x8eba87076d9dbeb6, rsp = 0xbbefedbfb8cb141b, rbp = 0x68194ad2f7128b89 ---

Tracing command kernel pid 0 tid 100017 td 0xfffffe0083858720 sched_switch() at sched_switch+0x818/frame 0xfffffe00836bce20 mi_switch() at mi_switch+0xc2/frame 0xfffffe00836bce40 _sleep() at _sleep+0x1ee/frame 0xfffffe00836bcec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00836bcef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00836bcf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00836bcf30 --- trap 0xc5e1fe4f, rip = 0x4c0b02ed4758ab3d, rsp = 0x5d77209d22aa20d9, rbp = 0x4556109e422ee1d1 ---

Tracing command kernel pid 0 tid 100019 td 0xfffffe00105fc740 sched_switch() at sched_switch+0x818/frame 0xfffffe000edf6e50 mi_switch() at mi_switch+0xc2/frame 0xfffffe000edf6e70 msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe000edf6ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xf4/frame 0xfffffe000edf6ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe000edf6f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000edf6f30 --- trap 0xdd86de81, rip = 0xc9c309b57079857f, rsp = 0x2bcacd08f85c9574, rbp = 0xf0fb7b0e949a4cba ---

Tracing command kernel pid 0 tid 100021 td 0xfffffe00105fb900 sched_switch() at sched_switch+0x818/frame 0xfffffe000edece20 mi_switch() at mi_switch+0xc2/frame 0xfffffe000edece40 _sleep() at _sleep+0x1ee/frame 0xfffffe000edecec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe000edecef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe000edecf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000edecf30 --- trap 0x48b875fd, rip = 0x575fcb32f5f78e63, rsp = 0xea0eb19000b264e9, rbp = 0xbd731f6b5d45eb27 ---

Tracing command kernel pid 0 tid 100023 td 0xfffffe00105faac0 sched_switch() at sched_switch+0x818/frame 0xfffffe000ede2e20 mi_switch() at mi_switch+0xc2/frame 0xfffffe000ede2e40 _sleep() at _sleep+0x1ee/frame 0xfffffe000ede2ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe000ede2ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe000ede2f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000ede2f30 --- trap 0xc646a901, rip = 0x6a11b1af00cd641f, rsp = 0xf3ea835f652ceed4, rbp = 0x6fc3ba0a17864e00 ---

Tracing command kernel pid 0 tid 100024 td 0xfffffe00105fa3a0 sched_switch() at sched_switch+0x818/frame 0xfffffe000eddde20 mi_switch() at mi_switch+0xc2/frame 0xfffffe000eddde40 _sleep() at _sleep+0x1ee/frame 0xfffffe000edddec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe000edddef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe000edddf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000edddf30 --- trap 0x3a1e31b0, rip = 0xb3f4fd9e98a754c2, rsp = 0xa288df57dd55df26, rbp = 0xbaa9efccbdd11e2e ---

Tracing command kernel pid 0 tid 100025 td 0xfffffe008385c020 sched_switch() at sched_switch+0x818/frame 0xfffffe00836e4e20 mi_switch() at mi_switch+0xc2/frame 0xfffffe00836e4e40 _sleep() at _sleep+0x1ee/frame 0xfffffe00836e4ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00836e4ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00836e4f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00836e4f30 --- trap 0x39b917fe, rip = 0x95af4e50ff325be8, rsp = 0xc167ca09af3502b, rbp = 0x903c45f5e879b57b ---

Tracing command kernel pid 0 tid 100026 td 0xfffffe008385b900 sched_switch() at sched_switch+0x818/frame 0xfffffe00836dfe20 mi_switch() at mi_switch+0xc2/frame 0xfffffe00836dfe40 _sleep() at _sleep+0x1ee/frame 0xfffffe00836dfec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00836dfef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00836dff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00836dff30 --- trap 0xd5d1ce2f, rip = 0x6ab94f74fd929ea9, rsp = 0x88bc8b5f75858e86, rbp = 0xcdf814c640bad9b ---

Tracing command kernel pid 0 tid 100027 td 0xfffffe008385b1e0 sched_switch() at sched_switch+0x818/frame 0xfffffe00836dae20 mi_switch() at mi_switch+0xc2/frame 0xfffffe00836dae40 _sleep() at _sleep+0x1ee/frame 0xfffffe00836daec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00836daef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00836daf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00836daf30 --- trap 0xc5e06e4f, rip = 0x4c0b02d96f58ab3d, rsp = 0x5d77209d22aa20d9, rbp = 0x455610de422ea1d1 ---

Tracing command kernel pid 0 tid 100028 td 0xfffffe008385aac0 sched_switch() at sched_switch+0x818/frame 0xfffffe00836d5e20 mi_switch() at mi_switch+0xc2/frame 0xfffffe00836d5e40 _sleep() at _sleep+0x1ee/frame 0xfffffe00836d5ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00836d5ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00836d5f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00836d5f30 --- trap 0x4b9ff6b3, rip = 0x714568a692624149, rsp = 0x4410125b4734cbc4, rbp = 0x97e6b5fd08ed7456 ---

Tracing command kernel pid 0 tid 100029 td 0xfffffe008385a3a0 sched_switch() at sched_switch+0x818/frame 0xfffffe00836d0e20 mi_switch() at mi_switch+0xc2/frame 0xfffffe00836d0e40 _sleep() at _sleep+0x1ee/frame 0xfffffe00836d0ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00836d0ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00836d0f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00836d0f30 --- trap 0x5bafd2d3, rip = 0x57f7358e08a874df, rsp = 0x91dbb9c4101bc5bb, rbp = 0xde6f26bd2ec8381c ---

Tracing command kernel pid 0 tid 100030 td 0xfffffe0083859c80 sched_switch() at sched_switch+0x818/frame 0xfffffe00836cbe20 mi_switch() at mi_switch+0xc2/frame 0xfffffe00836cbe40 _sleep() at _sleep+0x1ee/frame 0xfffffe00836cbec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00836cbef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00836cbf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00836cbf30 --- trap 0xee11f9ab, rip = 0xc96bf75a8d267bc1, rsp = 0x501fc593e8f5b466, rbp = 0x93f74246e317ab83 ---

Tracing command kernel pid 0 tid 100031 td 0xfffffe0083859560 sched_switch() at sched_switch+0x818/frame 0xfffffe00836c6e20 mi_switch() at mi_switch+0xc2/frame 0xfffffe00836c6e40 _sleep() at _sleep+0x1ee/frame 0xfffffe00836c6ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00836c6ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00836c6f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00836c6f30 --- trap 0x39b91afe, rip = 0x958a4e50ff325ee8, rsp = 0xc967ca09ad35023, rbp = 0x903c45f5e869b57f ---

Tracing command kernel pid 0 tid 100032 td 0xfffffe0083858e40 sched_switch() at sched_switch+0x818/frame 0xfffffe00836c1e20 mi_switch() at mi_switch+0xc2/frame 0xfffffe00836c1e40 _sleep() at _sleep+0x1ee/frame 0xfffffe00836c1ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00836c1ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00836c1f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00836c1f30 --- trap 0x70dfd533, rip = 0xd2f8c4a2e0d6ae21, rsp = 0x9cb35cebda44d104, rbp = 0x8ce74318bf1726e ---

Tracing command kernel pid 0 tid 100033 td 0xfffffe008385fac0 sched_switch() at sched_switch+0x818/frame 0xfffffe008370ce20 mi_switch() at mi_switch+0xc2/frame 0xfffffe008370ce40 _sleep() at _sleep+0x1ee/frame 0xfffffe008370cec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe008370cef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe008370cf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008370cf30 --- trap 0x2989349e, rip = 0xb3d4033565f8207c, rsp = 0xd95dd737cdfcfe70, rbp = 0xd9b5d6e7ce5cf935 ---

Tracing command kernel pid 0 tid 100034 td 0xfffffe008385f3a0 sched_switch() at sched_switch+0x818/frame 0xfffffe0083707e20 mi_switch() at mi_switch+0xc2/frame 0xfffffe0083707e40 _sleep() at _sleep+0x1ee/frame 0xfffffe0083707ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe0083707ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe0083707f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0083707f30 --- trap 0xb7c70a02, rip = 0xa82034cd0e08d19c, rsp = 0x15f14e6fff4d9917, rbp = 0x628ce094a2ba20f8 ---

Tracing command kernel pid 0 tid 100035 td 0xfffffe008385ec80 sched_switch() at sched_switch+0x818/frame 0xfffffe0083702e20 mi_switch() at mi_switch+0xc2/frame 0xfffffe0083702e40 _sleep() at _sleep+0x1ee/frame 0xfffffe0083702ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe0083702ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe0083702f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0083702f30 --- trap 0xa7e77a66, rip = 0x8e1239a890c24408, rsp = 0xc03ae5f8a8623549, rbp = 0xb057386849f6cb2 ---

Tracing command kernel pid 0 tid 100049 td 0xfffffe0083877000 sched_switch() at sched_switch+0x818/frame 0xfffffe0083778e20 mi_switch() at mi_switch+0xc2/frame 0xfffffe0083778e40 _sleep() at _sleep+0x1ee/frame 0xfffffe0083778ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe0083778ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe0083778f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0083778f30 --- trap 0x5bafd2d3, rip = 0x57f7358e08a874dd, rsp = 0x91dbb986101b45bb, rbp = 0xde6f26bd2ec8381c ---

Tracing command kernel pid 0 tid 100053 td 0xfffffe00838651e0 sched_switch() at sched_switch+0x818/frame 0xfffffe0083764e20 mi_switch() at mi_switch+0xc2/frame 0xfffffe0083764e40 _sleep() at _sleep+0x1ee/frame 0xfffffe0083764ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe0083764ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe0083764f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0083764f30 --- trap 0xaff7fa62, rip = 0x8e1279aab0c24708, rsp = 0xc03ae4f8a8623549, rbp = 0xb057386849f6cb0 ---

Tracing command kernel pid 0 tid 100054 td 0xfffffe0083864ac0 sched_switch() at sched_switch+0x818/frame 0xfffffe008375fe20 mi_switch() at mi_switch+0xc2/frame 0xfffffe008375fe40 _sleep() at _sleep+0x1ee/frame 0xfffffe008375fec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe008375fef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe008375ff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008375ff30 --- trap 0x4b9fb2b3, rip = 0x714578ae92420149, rsp = 0x4410125b4734ebe4, rbp = 0x97e6b5ed08ed7456 ---

Tracing command kernel pid 0 tid 100055 td 0xfffffe00838643a0 sched_switch() at sched_switch+0x818/frame 0xfffffe008375ae20 mi_switch() at mi_switch+0xc2/frame 0xfffffe008375ae40 _sleep() at _sleep+0x1ee/frame 0xfffffe008375aec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe008375aef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe008375af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008375af30 --- trap 0x5baed2d3, rip = 0x57f735ae00a874dd, rsp = 0x91dbb986101bc5bb, rbp = 0xde6f26bd2ec8381c ---

Tracing command kernel pid 0 tid 100056 td 0xfffffe0083863c80 sched_switch() at sched_switch+0x818/frame 0xfffffe0083755e20 mi_switch() at mi_switch+0xc2/frame 0xfffffe0083755e40 _sleep() at _sleep+0x1ee/frame 0xfffffe0083755ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe0083755ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe0083755f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0083755f30 --- trap 0xd591ce2f, rip = 0x6ab94f55fd929ea9, rsp = 0x88bc8b5f75858e84, rbp = 0xcdf83c4640bad9b ---

Tracing command kernel pid 0 tid 100074 td 0xfffffe008eb5c900 sched_switch() at sched_switch+0x818/frame 0xfffffe008f839e50 mi_switch() at mi_switch+0xc2/frame 0xfffffe008f839e70 msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe008f839ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xf4/frame 0xfffffe008f839ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe008f839f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008f839f30 --- trap 0x9e0020, rip = 0x1000008000033120, rsp = 0xfa7f99db8225c15a, rbp = 0x25002a8e1cc90000 ---

Tracing command kernel pid 0 tid 100075 td 0xfffffe008eb5c1e0 sched_switch() at sched_switch+0x818/frame 0xfffffe008f834e50 mi_switch() at mi_switch+0xc2/frame 0xfffffe008f834e70 msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe008f834ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xf4/frame 0xfffffe008f834ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe008f834f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008f834f30 --- trap 0x72c100, rip = 0x4c80bcca80061080, rsp = 0x2efffafe3213411a, rbp = 0x2640291831830208 ---

Tracing command kernel pid 0 tid 100076 td 0xfffffe008eb5bac0 sched_switch() at sched_switch+0x818/frame 0xfffffe008f82fe50 mi_switch() at mi_switch+0xc2/frame 0xfffffe008f82fe70 msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe008f82fec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xf4/frame 0xfffffe008f82fef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe008f82ff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008f82ff30 --- trap 0x4388a018, rip = 0xd23f3a0112814a8e, rsp = 0xe76654ffca3bd21a, rbp = 0x6bc244c6f01d8015 ---

Tracing command kernel pid 0 tid 100077 td 0xfffffe008eb5b3a0 sched_switch() at sched_switch+0x818/frame 0xfffffe008f82ae50 mi_switch() at mi_switch+0xc2/frame 0xfffffe008f82ae70 msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe008f82aec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xf4/frame 0xfffffe008f82aef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe008f82af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008f82af30 --- trap 0x1200c25d, rip = 0xf48d716084012e0b, rsp = 0x32adff6e9f225b1b, rbp = 0x224bce015a491116 ---

Tracing command kernel pid 0 tid 100079 td 0xfffffe008eb5a560 sched_switch() at sched_switch+0x818/frame 0xfffffe008f820e20 mi_switch() at mi_switch+0xc2/frame 0xfffffe008f820e40 _sleep() at _sleep+0x1ee/frame 0xfffffe008f820ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe008f820ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe008f820f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008f820f30 --- trap 0xc9b46ca1, rip = 0xef7146b96ab3b2db, rsp = 0xfe0176b9ae6b997b, rbp = 0xb972b8bfb6bf1ad0 ---

Tracing command kernel pid 0 tid 100092 td 0xfffffe00838d61e0 sched_switch() at sched_switch+0x818/frame 0xfffffe00837d6e20 mi_switch() at mi_switch+0xc2/frame 0xfffffe00837d6e40 _sleep() at _sleep+0x1ee/frame 0xfffffe00837d6ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00837d6ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00837d6f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00837d6f30 --- trap 0x4b9ff2b3, rip = 0x71457871d2624349, rsp = 0x441012534734ebe4, rbp = 0x97e6b5bf08ed7456 ---

Tracing command init pid 1 tid 100002 td 0xfffffe00105cdac0 sched_switch() at sched_switch+0x818/frame 0xfffffe000edaba70 mi_switch() at mi_switch+0xc2/frame 0xfffffe000edaba90 sleepq_catch_signals() at sleepq_catch_signals+0x2e6/frame 0xfffffe000edabae0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe000edabaf0 _sleep() at _sleep+0x1e1/frame 0xfffffe000edabb70 kern_wait6() at kern_wait6+0x560/frame 0xfffffe000edabc00 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe000edabe00 amd64_syscall() at amd64_syscall+0x10c/frame 0xfffffe000edabf30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe000edabf30 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x26f05a, rsp = 0x820c3f888, rbp = 0x820c3f8f0 ---

Tracing command KTLS pid 2 tid 100036 td 0xfffffe008385e560 sched_switch() at sched_switch+0x818/frame 0xfffffe00836fdd50 mi_switch() at mi_switch+0xc2/frame 0xfffffe00836fdd70 _sleep() at _sleep+0x1ee/frame 0xfffffe00836fddf0 ktls_work_thread() at ktls_work_thread+0xe9/frame 0xfffffe00836fdef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00836fdf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00836fdf30 --- trap 0xd5d1ce2f, rip = 0x6ab94f54fd929ea9, rsp = 0x88bc8b5d75858e06, rbp = 0xcdf8344640bad9b ---

Tracing command KTLS pid 2 tid 100037 td 0xfffffe008385de40 sched_switch() at sched_switch+0x818/frame 0xfffffe00836f8d50 mi_switch() at mi_switch+0xc2/frame 0xfffffe00836f8d70 _sleep() at _sleep+0x1ee/frame 0xfffffe00836f8df0 ktls_work_thread() at ktls_work_thread+0xe9/frame 0xfffffe00836f8ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00836f8f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00836f8f30 --- trap 0xc5e1ee5f, rip = 0x4c0b02992758ab3d, rsp = 0x5d7720b922a220d9, rbp = 0x455600de422ee1d1 ---

Tracing command KTLS pid 2 tid 100038 td 0xfffffe008385d720 sched_switch() at sched_switch+0x818/frame 0xfffffe00836f3d50 mi_switch() at mi_switch+0xc2/frame 0xfffffe00836f3d70 _sleep() at _sleep+0x1ee/frame 0xfffffe00836f3df0 ktls_work_thread() at ktls_work_thread+0xe9/frame 0xfffffe00836f3ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00836f3f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00836f3f30 --- trap 0x5bafc3d3, rip = 0x57f7357308a87559, rsp = 0x91dbb9c4101945ba, rbp = 0xde6f26ad2ec83a1c ---

Tracing command KTLS pid 2 tid 100039 td 0xfffffe008385d000 sched_switch() at sched_switch+0x818/frame 0xfffffe00836eed50 mi_switch() at mi_switch+0xc2/frame 0xfffffe00836eed70 _sleep() at _sleep+0x1ee/frame 0xfffffe00836eedf0 ktls_work_thread() at ktls_work_thread+0xe9/frame 0xfffffe00836eeef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00836eef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00836eef30 --- trap 0xa7f72a62, rip = 0x8e1679a890c24608, rsp = 0xc03ae5f8a8622549, rbp = 0xb056386849f6cba ---

Tracing command crypto pid 3 tid 100057 td 0xfffffe008387a900 sched_switch() at sched_switch+0x818/frame 0xfffffe00837a0e10 mi_switch() at mi_switch+0xc2/frame 0xfffffe00837a0e30 _sleep() at _sleep+0x1ee/frame 0xfffffe00837a0eb0 crypto_dispatch_thread() at crypto_dispatch_thread+0x21a/frame 0xfffffe00837a0ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00837a0f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00837a0f30 --- trap 0x406fe553, rip = 0xf43f8dc7381cb5b5, rsp = 0x4978f77c8d6b7f4f, rbp = 0x4155e723add43e24 ---

Tracing command crypto pid 3 tid 100058 td 0xfffffe008387a1e0 sched_switch() at sched_switch+0x818/frame 0xfffffe008379bde0 mi_switch() at mi_switch+0xc2/frame 0xfffffe008379be00 _sleep() at _sleep+0x1ee/frame 0xfffffe008379be80 crypto_ret_thread() at crypto_ret_thread+0x1a3/frame 0xfffffe008379bef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe008379bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008379bf30 --- trap 0xac263d86, rip = 0xb728c5b3abc90f4, rsp = 0xcd520055623da1f6, rbp = 0xddb4211221a626c0 ---

Tracing command crypto pid 3 tid 100059 td 0xfffffe0083879ac0 sched_switch() at sched_switch+0x818/frame 0xfffffe0083796de0 mi_switch() at mi_switch+0xc2/frame 0xfffffe0083796e00 _sleep() at _sleep+0x1ee/frame 0xfffffe0083796e80 crypto_ret_thread() at crypto_ret_thread+0x1a3/frame 0xfffffe0083796ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe0083796f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0083796f30 --- trap 0x5bafd2d3, rip = 0x57f7354e08a874dd, rsp = 0x91dbb986101b45bb, rbp = 0xde6f26bd2ec8381c ---

Tracing command crypto pid 3 tid 100060 td 0xfffffe00838793a0 sched_switch() at sched_switch+0x818/frame 0xfffffe0083791de0 mi_switch() at mi_switch+0xc2/frame 0xfffffe0083791e00 _sleep() at _sleep+0x1ee/frame 0xfffffe0083791e80 crypto_ret_thread() at crypto_ret_thread+0x1a3/frame 0xfffffe0083791ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe0083791f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0083791f30 --- trap 0x3249011e, rip = 0x108ebb56154c4f14, rsp = 0x1fe99f2508cc494, rbp = 0x468d17694d40ff0d ---

Tracing command crypto pid 3 tid 100061 td 0xfffffe0083878c80 sched_switch() at sched_switch+0x818/frame 0xfffffe008378cde0 mi_switch() at mi_switch+0xc2/frame 0xfffffe008378ce00 _sleep() at _sleep+0x1ee/frame 0xfffffe008378ce80 crypto_ret_thread() at crypto_ret_thread+0x1a3/frame 0xfffffe008378cef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe008378cf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008378cf30 --- trap 0xce11f9af, rip = 0xc969f75a8d263ec1, rsp = 0x501fc5b3e8f5b426, rbp = 0x93e74242e717aba3 ---

Tracing command cam pid 4 tid 100063 td 0xfffffe0083877e40 sched_switch() at sched_switch+0x818/frame 0xfffffe0083782e00 mi_switch() at mi_switch+0xc2/frame 0xfffffe0083782e20 _sleep() at _sleep+0x1ee/frame 0xfffffe0083782ea0 xpt_done_td() at xpt_done_td+0x8b/frame 0xfffffe0083782ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe0083782f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0083782f30 --- trap 0x406fe553, rip = 0xf4378dcf781c95b5, rsp = 0x4978f77c8d6b7f5f, rbp = 0x4157e723add43e24 ---

Tracing command cam pid 4 tid 100080 td 0xfffffe008f7f11e0 sched_switch() at sched_switch+0x818/frame 0xfffffe008f86fe10 mi_switch() at mi_switch+0xc2/frame 0xfffffe008f86fe30 _sleep() at _sleep+0x1ee/frame 0xfffffe008f86feb0 xpt_scanner_thread() at xpt_scanner_thread+0x88/frame 0xfffffe008f86fef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe008f86ff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008f86ff30 --- trap 0x41720040, rip = 0xf48d77ff871bc71b, rsp = 0x32ad7f0001401001, rbp = 0x224bd2844a510010 ---

Tracing command pf purge pid 5 tid 100071 td 0xfffffe0083877720 sched_switch() at sched_switch+0x818/frame 0xfffffe008377dde0 mi_switch() at mi_switch+0xc2/frame 0xfffffe008377de00 sleepq_timedwait() at sleepq_timedwait+0x2f/frame 0xfffffe008377de40 _sleep() at _sleep+0x1ce/frame 0xfffffe008377dec0 pf_purge_thread() at pf_purge_thread+0xb6/frame 0xfffffe008377def0 fork_exit() at fork_exit+0x7e/frame 0xfffffe008377df30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008377df30 --- trap 0x498cd2b3, rip = 0x7145783196744b2b, rsp = 0x441012114754ebe4, rbp = 0x97e6b53328ed7c56 ---

Tracing command rand_harvestq pid 6 tid 100078 td 0xfffffe008eb5ac80 sched_switch() at sched_switch+0x818/frame 0xfffffe008f825d80 mi_switch() at mi_switch+0xc2/frame 0xfffffe008f825da0 sleepq_timedwait() at sleepq_timedwait+0x2f/frame 0xfffffe008f825de0 _sleep() at _sleep+0x1ce/frame 0xfffffe008f825e60 random_kthread() at random_kthread+0xa7/frame 0xfffffe008f825ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe008f825f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008f825f30 --- trap 0xdd049881, rip = 0xc9c329917075c97a, rsp = 0x2bcac5bbd898e534, rbp = 0xf0fb7b089d9248b8 ---

Tracing command pagedaemon pid 7 tid 100081 td 0xfffffe008f7f0ac0 sched_switch() at sched_switch+0x818/frame 0xfffffe008f86ab10 mi_switch() at mi_switch+0xc2/frame 0xfffffe008f86ab30 sleepq_timedwait() at sleepq_timedwait+0x2f/frame 0xfffffe008f86ab70 _sleep() at _sleep+0x1ce/frame 0xfffffe008f86abf0 vm_pageout_worker() at vm_pageout_worker+0x28c/frame 0xfffffe008f86aeb0 vm_pageout() at vm_pageout+0x1d7/frame 0xfffffe008f86aef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe008f86af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008f86af30 --- trap 0x288201d, rip = 0xd23f08080d81789f, rsp = 0xe766101c2b7f84d0, rbp = 0x6bc2490a30549854 ---

Tracing command pagedaemon pid 7 tid 100084 td 0xfffffe00838d4c80 sched_switch() at sched_switch+0x818/frame 0xfffffe00837c3bd0 mi_switch() at mi_switch+0xc2/frame 0xfffffe00837c3bf0 _sleep() at _sleep+0x1ee/frame 0xfffffe00837c3c70 vm_pageout_laundry_worker() at vm_pageout_laundry_worker+0xcd0/frame 0xfffffe00837c3ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00837c3f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00837c3f30 --- trap 0x101e0008, rip = 0xd281c00a24974848, rsp = 0x8c0008ab3268aa08, rbp = 0x8080409a039aa60 ---

Tracing command pagedaemon pid 7 tid 100085 td 0xfffffe00838d4560 sched_switch() at sched_switch+0x818/frame 0xfffffe00837bedd0 mi_switch() at mi_switch+0xc2/frame 0xfffffe00837bedf0 sleepq_timedwait() at sleepq_timedwait+0x2f/frame 0xfffffe00837bee30 _sleep() at _sleep+0x1ce/frame 0xfffffe00837beeb0 uma_reclaim_worker() at uma_reclaim_worker+0xe8/frame 0xfffffe00837beef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00837bef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00837bef30 --- trap 0xc5412a4f, rip = 0x4c0b0208275a3b18, rsp = 0x5d77000822bfb498, rbp = 0x4556100c023ff0c1 ---

Tracing command vmdaemon pid 8 tid 100082 td 0xfffffe008f7f03a0 sched_switch() at sched_switch+0x818/frame 0xfffffe008f865de0 mi_switch() at mi_switch+0xc2/frame 0xfffffe008f865e00 _sleep() at _sleep+0x1ee/frame 0xfffffe008f865e80 vm_daemon() at vm_daemon+0x96/frame 0xfffffe008f865ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe008f865f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008f865f30 --- trap 0x4c340e14, rip = 0xef7144004aa3b44b, rsp = 0xfe016408cf123f78, rbp = 0xb972e002b61622d2 ---

Tracing command bufdaemon pid 9 tid 100083 td 0xfffffe008f7efconfig.txt0600001106514472620143 7541 ustarrootwheeloptions CONFIG_AUTOGENERATED ident SMP machine amd64 cpu HAMMER

WARNING: / was not properly dismounted WARNING: /: mount pending error: blocks 24 files 1 <118>Mounting filesystems... <118>tunefs: / is not clean - run fsck. <118> <118>tunefs: / is not clean - run fsck. <118> <118>** /dev/ufs/OPNsense <118>** Last Mounted on /tmp/hdrescue <118>** Root file system <118>** Phase 1 - Check Blocks and Sizes <118>INCORRECT BLOCK COUNT I=802571 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802606 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802618 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802642 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802662 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802663 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802666 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802672 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802698 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802704 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802724 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802725 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802730 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802734 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802743 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802756 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802758 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802761 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802769 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802770 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802775 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802776 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802778 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802779 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802780 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802781 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802786 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802787 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802792 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802795 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802796 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802797 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802798 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802800 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802805 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802808 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802812 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802813 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802816 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802823 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802825 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802830 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802831 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802841 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802889 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802893 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802899 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802916 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802936 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802937 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802946 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802951 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=802953 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803013 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803016 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803069 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803073 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803075 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803093 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803094 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803103 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803176 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803181 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803189 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803195 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803196 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803197 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803198 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803199 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803200 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803201 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803202 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803203 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803211 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803212 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803213 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803214 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803215 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803216 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803217 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803218 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803219 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803220 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803221 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803222 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803260 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803261 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803267 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803303 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803304 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803319 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803320 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803328 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803600 (448 should be 320) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803601 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803631 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803632 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803633 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803634 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803635 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803636 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803639 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803640 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=803643 (24 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=804090 (8 should be 0) <118>CORRECT? yes <118> <118>INCORRECT BLOCK COUNT I=882831 (8 should be 0) <118>CORRECT? yes <118> <118>** Phase 2 - Check Pathnames <118>** Phase 3 - Check Connectivity <118>** Phase 4 - Check Reference Counts <118>** Phase 5 - Check Cyl groups <118>FREE BLK COUNT(S) WRONG IN SUPERBLK <118>SALVAGE? yes <118> <118>SUMMARY INFORMATION BAD <118>SALVAGE? yes <118> <118>BLK(S) MISSING IN BIT MAPS <118>SALVAGE? yes <118> <118>88481 files, 2446402 used, 8627633 free (20577 frags, 1075882 blocks, 0.2% fragmentation) <118> <118>***** FILE SYSTEM MARKED CLEAN ***** <118> <118>***** FILE SYSTEM WAS MODIFIED *****

Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x0 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff823896cf stack pointer = 0x28:0xfffffe008373bb30 frame pointer = 0x28:0xfffffe008373bb90 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 12 (swi1: netisr 2) trap number = 12 panic: page fault cpuid = 0 time = 1693130851 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe008373b8f0 vpanic() at vpanic+0x151/frame 0xfffffe008373b940 panic() at panic+0x43/frame 0xfffffe008373b9a0 trap_fatal() at trap_fatal+0x387/frame 0xfffffe008373ba00 trap_pfault() at trap_pfault+0x4f/frame 0xfffffe008373ba60 calltrap() at calltrap+0x8/frame 0xfffffe008373ba60 --- trap 0xc, rip = 0xffffffff823896cf, rsp = 0xfffffe008373bb30, rbp = 0xfffffe008373bb90 --- pf_test_state_udp() at pf_test_state_udp+0x28f/frame 0xfffffe008373bb90 pf_test() at pf_test+0xc57/frame 0xfffffe008373bd00 pf_check_in() at pf_check_in+0x25/frame 0xfffffe008373bd20 pfil_run_hooks() at pfil_run_hooks+0x97/frame 0xfffffe008373bd60 ip_input() at ip_input+0x799/frame 0xfffffe008373bdf0 swi_net() at swi_net+0x12b/frame 0xfffffe008373be60 ithread_loop() at ithread_loop+0x25a/frame 0xfffffe008373bef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe008373bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008373bf30 --- trap 0x3249811e, rip = 0x108ebb17154ccb14, rsp = 0x1fe99f2508cc494, rbp = 0x468d176d4d40ff0d --- KDB: enter: panic panic.txt0600001214472620143 7134 ustarrootwheelpage faultversion.txt0600007414472620143 7537 ustarrootwheelFreeBSD 13.2-RELEASE-p2 stable/23.7-n254761-4b4f06e3731 SMP

Additional context

Add any other context about the problem here.

Environment

Software version used and hardware type if relevant, e.g.:

OPNsense 23.7.1-amd64 - Current stable FreeBSD 13.2-RELEASE-p2 OpenSSL 1.1.1v 1 Aug 2023 Whitebox Mini PC w/Passive Cooling, 4 NIC, 1 COM, VGA, USB Intel(R) Celeron(R) CPU J1900 @ 1.99GHz (4 cores, 4 threads) 8GB RAM 64GB SSD Intel 82583V NICs (4-port)

kb8080 avatar Aug 28 '23 16:08 kb8080

pf_test_state_udp() is a bit suspicious here... do you have a HA setup?

Cheers, Franco

fichtner avatar Aug 29 '23 12:08 fichtner

For reference:

https://forum.opnsense.org/index.php?topic=34348.0 https://forum.opnsense.org/index.php?topic=35597.0

fichtner avatar Aug 29 '23 12:08 fichtner

Thanks for the response and links. I don't have a HA setup in my case. When I did my search of the forum this past weekend I didn't see that second link. Seems to have very similar behavior to what I'm seeing although in my case I'm bare metal not an Unraid VM. Did the upstream Free BSD kernel change from 23.7.1 > 23.7.2 that may have introduced the problem?

kb8080 avatar Aug 29 '23 13:08 kb8080

Nothing that would directly explain this, no. If I had to take a bet it would be FreeBSD 13.2 itself in 23.7, but it's not quite clear yet.

fichtner avatar Aug 29 '23 13:08 fichtner

Ok. Not sure how to proceed at this point but happy to help troubleshoot. I have a spare firewall I can use to keep me operational so if debugging the kernel or otherwise is in order and you're willing to provide instruction, I'm open to help (if it helps).

kb8080 avatar Aug 29 '23 13:08 kb8080

There is two options: try to debug with a vanilla FreeBSD 13.2 kernel or try to debug on an OPNsense debug kernel first. The first is preferred by upstream for creating reports, but it could introduce issues either already fixed by us or features breaking due to missing upstream support (we do have a separate kernel for these good reasons).

fichtner avatar Aug 29 '23 13:08 fichtner

Closing, no response.

fichtner avatar Apr 04 '24 15:04 fichtner

Could have been 7313147a28e

fichtner avatar May 23 '24 07:05 fichtner