operating-system icon indicating copy to clipboard operation
operating-system copied to clipboard
verified publisher icon home-assistant

HA OS 16.3 shows permanently segfaulting programs and processes and is unstable

Open tzdisco4 opened this issue 1 month ago • 4 comments

Describe the issue you are experiencing

After upgrading HA OS from 16.2 to 16.3 became unstable after months of stable operation. I already reported some experiences in this issue: https://github.com/home-assistant/operating-system/issues/4407

The issues (segfaults, core crashes, reboot, blocked system during HA OS reboot, systemd segfault during shutdown, ...) persisted. So I decicded yesterday to install a new vbox engine (on Ubuntu, I5 nuc) from scratch: https://github.com/home-assistant/operating-system/releases/download/16.3/haos_ova-16.3.vdi.zip

After restoring my backup and some tweaking HA was up and running again. But I now see in the system journal the very same programs and services segfaulting. This should not happen. I can imagine that some of the shared object files not the ones which were used to link the executables.

Anyway, here the list of segfaults since the last ~20 hours:

# journalctl|grep segfault
Nov 30 18:30:39 homeassistant kernel: nc[42682]: segfault at 7f8700000004 ip 00007f877b7a6159 sp 00007ffc78d29a18 error 4 in libc.so.6[15c159,7f877b672000+15a000] likely on CPU 1 (core 1, socket 0)
Nov 30 20:05:25 homeassistant kernel: curl[69739]: segfault at 56270000114c ip 00007f6db45f1769 sp 00007ffd4c5e30f8 error 4 in ld-musl-x86_64.so.1[24769,7f6db45e1000+54000] likely on CPU 0 (core 0, socket 0)
Nov 30 20:27:30 homeassistant kernel: nc[75768]: segfault at 7fcb0000115c ip 00007fcb3db85662 sp 00007ffc60bc0670 error 4 in libcrypto.so.3[10a662,7fcb3db65000+335000] likely on CPU 0 (core 0, socket 0)
Nov 30 20:30:15 homeassistant kernel: tclsh[76470]: segfault at 228c ip 00007ff8fcc1514c sp 00007ffdfe5f3550 error 4 in libtcl8.6.so[10714c,7ff8fcb39000+142000] likely on CPU 0 (core 0, socket 0)
Nov 30 21:46:47 homeassistant kernel: curl[97420]: segfault at 55a700002284 ip 00007ff09ffe3769 sp 00007ffc80df6288 error 4 in ld-musl-x86_64.so.1[24769,7ff09ffd3000+54000] likely on CPU 0 (core 0, socket 0)
Nov 30 21:48:18 homeassistant kernel: curl[97799]: segfault at 55f9000033bc ip 00007f4d3c067769 sp 00007ffde02885c8 error 4 in ld-musl-x86_64.so.1[24769,7f4d3c057000+54000] likely on CPU 1 (core 1, socket 0)
Nov 30 22:23:15 homeassistant kernel: tclsh[107352]: segfault at 33ac ip 00007f3c6218839a sp 00007ffff6d439f0 error 4 in libtcl8.6.so[11639a,7f3c6209d000+142000] likely on CPU 0 (core 0, socket 0)
Dec 01 01:36:15 homeassistant kernel: git-remote-http[160887]: segfault at 7f92000033cc ip 00007f92b54a8f87 sp 00007fffb7958f00 error 4 in libcrypto.so.3[6ff87,7f92b548a000+2d1000] likely on CPU 1 (core 1, socket 0)
Dec 01 01:36:16 homeassistant kernel: git-remote-http[160953]: segfault at 7f0000000024 ip 00007f0037530f87 sp 00007ffef996dde0 error 4 in libcrypto.so.3[6ff87,7f0037512000+2d1000] likely on CPU 1 (core 1, socket 0)
Dec 01 02:58:46 homeassistant kernel: tclsh[183865]: segfault at 1154 ip 00007f10da38214c sp 00007ffd238d2610 error 4 in libtcl8.6.so[10714c,7f10da2a6000+142000] likely on CPU 0 (core 0, socket 0)
Dec 01 03:01:16 homeassistant kernel: tclsh[184592]: segfault at 560e000033b4 ip 00007f5749fd8827 sp 00007ffc2af18550 error 4 in libtcl8.6.so[42827,7f5749fc1000+142000] likely on CPU 1 (core 1, socket 0)
Dec 01 03:07:03 homeassistant kernel: grep[186189]: segfault at 7f1f0000002c ip 00007f1fe241b237 sp 00007ffe07c3eaa0 error 4 in ld-linux-x86-64.so.2[a237,7f1fe2412000+28000] likely on CPU 0 (core 0, socket 0)
Dec 01 03:07:05 homeassistant kernel: checkCoProcesso[186207]: segfault at 7f3c0000002c ip 00007f3cbb392237 sp 00007fffb56b31a0 error 4 in ld-linux-x86-64.so.2[a237,7f3cbb389000+28000] likely on CPU 0 (core 0, socket 0)
Dec 01 03:07:05 homeassistant kernel: checkCoProcesso[186205]: segfault at 7f3c0000229c ip 00007f3cbb392237 sp 00007fffb56b3060 error 4 in ld-linux-x86-64.so.2[a237,7f3cbb389000+28000] likely on CPU 1 (core 1, socket 0)
Dec 01 07:36:20 homeassistant kernel: git-remote-http[259411]: segfault at 7f580000115c ip 00007f58be772f87 sp 00007fff6eb8f8e0 error 4 in libcrypto.so.3[6ff87,7f58be754000+2d1000] likely on CPU 0 (core 0, socket 0)
Dec 01 07:48:02 homeassistant kernel: awk[262902]: segfault at 7fdb0000035a ip 00007fdb5e96bb03 sp 00007fff428cc800 error 4 in ld-linux-x86-64.so.2[5b03,7fdb5e967000+28000] likely on CPU 0 (core 0, socket 0)
Dec 01 07:49:03 homeassistant kernel: curl[263200]: segfault at 562b000033bc ip 00007f25b34e7769 sp 00007fff4c8619a8 error 4 in ld-musl-x86_64.so.1[24769,7f25b34d7000+54000] likely on CPU 1 (core 1, socket 0)
Dec 01 07:49:50 homeassistant kernel: find[263460]: segfault at 7fd9000ede4c ip 00007fd95ce2afc0 sp 00007fff30e50e30 error 4 in ld-linux-x86-64.so.2[1fc0,7fd95ce2a000+28000] likely on CPU 0 (core 0, socket 0)

I attach the complete journal.

journal.tar.gz

What operating system image do you use?

ova (for Virtual Machines)

What version of Home Assistant Operating System is installed?

Home Assistant OS 16.3

Did the problem occur after upgrading the Operating System?

Yes

Hardware details

HA runs in a VirtualBox 7.0.16_Ubuntu r162802 on an Intel NUC, 256 G ssd, 8G ram, I5 CPU The host shows nor issues in it's logs.

tz@smart-nuc:~$ uname -a Linux smart-nuc 6.14.0-36-generic #36~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Oct 15 15:45:17 UTC 2 x86_64 x86_64 x86_64 GNU/Linux tz@smart-nuc:~$ lsusb Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 002: ID 1b1f:c020 eQ-3 Entwicklung GmbH HmIP-RFUSB Bus 001 Device 003: ID 8087:0a2b Intel Corp. Bluetooth wireless interface Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

Steps to reproduce the issue

1.Upgrade from HA OS 16.2 to 16.3 on an OVA image. 2. 3. ...

Anything in the Supervisor logs that might be useful for us?

[32m2025-12-01 07:34:16.489 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state running[0m
[32m2025-12-01 07:34:16.489 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete[0m
[32m2025-12-01 07:36:19.123 INFO (MainThread) [supervisor.store.git] Update add-on /data/addons/git/5c53de3b repository from https://github.com/esphome/home-assistant-addon[0m
[32m2025-12-01 07:36:19.124 INFO (MainThread) [supervisor.store.git] Update add-on /data/addons/git/a0d7b954 repository from https://github.com/hassio-addons/repository[0m
[32m2025-12-01 07:36:19.138 INFO (MainThread) [supervisor.store.git] Update add-on /data/addons/core repository from https://github.com/home-assistant/addons[0m
[32m2025-12-01 07:36:19.139 INFO (MainThread) [supervisor.store.git] Update add-on /data/addons/git/d5369777 repository from https://github.com/music-assistant/home-assistant-addon[0m
[32m2025-12-01 07:36:19.149 INFO (MainThread) [supervisor.store.git] Update add-on /data/addons/git/6a2c1c7a repository from https://github.com/kellerza/hassio-sma-em[0m
[32m2025-12-01 07:36:19.152 INFO (MainThread) [supervisor.store.git] Update add-on /data/addons/git/de838cd8 repository from https://github.com/jens-maus/RaspberryMatic[0m
[32m2025-12-01 07:36:19.154 INFO (MainThread) [supervisor.store.git] Update add-on /data/addons/git/bc3b947b repository from https://github.com/Akulatraxas/ha-modbusproxy[0m
[31m2025-12-01 07:36:20.485 ERROR (MainThread) [supervisor.store.git] Can't update https://github.com/jens-maus/RaspberryMatic repo: Cmd('git') failed due to: exit code(1)
  cmdline: git fetch -v --update-shallow --depth=1 -- origin
  stderr: 'error: git-remote-https died of signal 11'.[0m
[32m2025-12-01 07:36:20.485 INFO (MainThread) [supervisor.resolution.module] Create new suggestion execute_reset - store / de838cd8[0m
[32m2025-12-01 07:36:20.485 INFO (MainThread) [supervisor.resolution.module] Create new issue corrupt_repository - store / de838cd8[0m
[31m2025-12-01 07:36:20.878 ERROR (MainThread) [supervisor.store] Could not reload repository de838cd8 due to StoreGitError()[0m
[32m2025-12-01 07:36:21.999 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token[0m
[33m2025-12-01 07:36:21.942 WARNING (SyncWorker_5) [supervisor.addons.validate] Add-on 'Overseerr' uses deprecated 'codenotary' field in config. This field is no longer used and will be ignored. Please report this to the maintainer.[0m
[33m2025-12-01 07:36:21.945 WARNING (SyncWorker_5) [supervisor.addons.validate] Add-on 'InfluxDB' uses deprecated 'codenotary' field in config. This field is no longer used and will be ignored. Please report this to the maintainer.[0m
[33m2025-12-01 07:36:21.947 WARNING (SyncWorker_5) [supervisor.addons.validate] Add-on 'MQTT IO' uses deprecated 'codenotary' field in config. This field is no longer used and will be ignored. Please report this to the maintainer.[0m
[33m2025-12-01 07:36:21.960 WARNING (SyncWorker_5) [supervisor.addons.validate] Add-on 'Tailscale' uses deprecated 'codenotary' field in config. This field is no longer used and will be ignored. Please report this to the maintainer.[0m
[33m2025-12-01 07:36:21.971 WARNING (SyncWorker_5) [supervisor.addons.validate] Add-on 'Log Viewer' uses deprecated 'codenotary' field in config. This field is no longer used and will be ignored. Please report this to the maintainer.[0m
[33m2025-12-01 07:36:22.999 WARNING (SyncWorker_5) [supervisor.addons.validate] Add-on 'Glances' uses deprecated 'codenotary' field in config. This field is no longer used and will be ignored. Please report this to the maintainer.[0m
[33m2025-12-01 07:36:22.002 WARNING (SyncWorker_5) [supervisor.addons.validate] Add-on 'Grocy' uses deprecated 'codenotary' field in config. This field is no longer used and will be ignored. Please report this to the maintainer.[0m
[33m2025-12-01 07:36:22.015 WARNING (SyncWorker_5) [supervisor.addons.validate] Add-on 'Nginx Proxy Manager' uses deprecated 'codenotary' field in config. This field is no longer used and will be ignored. Please report this to the maintainer.[0m
[33m2025-12-01 07:36:22.017 WARNING (SyncWorker_5) [supervisor.addons.validate] Add-on 'Folding@home' uses deprecated 'codenotary' field in config. This field is no longer used and will be ignored. Please report this to the maintainer.[0m
[33m2025-12-01 07:36:22.024 WARNING (SyncWorker_5) [supervisor.addons.validate] Add-on 'Traccar' uses deprecated 'codenotary' field in config. This field is no longer used and will be ignored. Please report this to the maintainer.[0m
[32m2025-12-01 07:36:22.065 INFO (MainThread) [supervisor.store] Loading add-ons from store: 88 all - 0 new - 0 remove[0m
[32m2025-12-01 07:39:31.878 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json[0m
[32m2025-12-01 08:06:21.342 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token[0m
[32m2025-12-01 08:34:16.500 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state running[0m
[32m2025-12-01 08:34:16.501 INFO (MainThread) [supervisor.resolution.checks.base] Run check for multiple_data_disks/system[0m
[32m2025-12-01 08:34:16.502 INFO (MainThread) [supervisor.resolution.checks.base] Run check for ipv4_connection_problem/system[0m
[32m2025-12-01 08:34:16.503 INFO (MainThread) [supervisor.resolution.checks.base] Run check for disk_lifetime/system[0m
[32m2025-12-01 08:34:16.511 INFO (MainThread) [supervisor.resolution.checks.base] Run check for disabled_data_disk/system[0m
[32m2025-12-01 08:34:16.511 INFO (MainThread) [supervisor.resolution.checks.base] Run check for pwned/addon[0m
[32m2025-12-01 08:34:16.651 INFO (MainThread) [supervisor.resolution.checks.base] Run check for security/core[0m
[32m2025-12-01 08:34:16.652 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_ipv6_error/dns_server[0m
[32m2025-12-01 08:34:16.655 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_failed/dns_server[0m
[32m2025-12-01 08:34:16.656 INFO (MainThread) [supervisor.resolution.checks.base] Run check for docker_config/system[0m
[32m2025-12-01 08:34:16.657 INFO (MainThread) [supervisor.resolution.checks.base] Run check for free_space/system[0m
[32m2025-12-01 08:34:16.661 INFO (MainThread) [supervisor.resolution.check] System checks complete[0m
[32m2025-12-01 08:34:16.661 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state running[0m
[32m2025-12-01 08:34:16.743 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete[0m
[32m2025-12-01 08:34:16.743 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state running[0m
[32m2025-12-01 08:34:16.745 INFO (MainThread) [supervisor.resolution.fixups.store_execute_reset] Reset corrupt Store: de838cd8[0m
[32m2025-12-01 08:34:16.746 INFO (MainThread) [supervisor.store.git] Cloning add-on /data/tmp/tmp27ghnq55 repository from https://github.com/jens-maus/RaspberryMatic[0m
[31m2025-12-01 08:34:20.408 ERROR (MainThread) [supervisor.store.git] Can't clone https://github.com/jens-maus/RaspberryMatic repository: Cmd('git') failed due to: exit code(128)
  cmdline: git clone -v --recursive --depth=1 --shallow-submodules -- https://github.com/jens-maus/RaspberryMatic /data/tmp/tmp27ghnq55
  stderr: 'Cloning into '/data/tmp/tmp27ghnq55'...
POST git-upload-pack (181 bytes)
POST git-upload-pack (250 bytes)
fatal: serious inflate inconsistency
fatal: fetch-pack: invalid index-pack output
'

Anything in the Host logs that might be useful for us?

Yes, the segfaults. Please look at the attached journal.

System information

## System Information

version | core-2025.11.3
-- | --
installation_type | Home Assistant OS
dev | false
hassio | true
docker | true
container_arch | amd64
user | root
virtualenv | false
python_version | 3.13.9
os_name | Linux
os_version | 6.12.51-haos
arch | x86_64
timezone | Europe/Berlin
config_dir | /config

<details><summary>Home Assistant Community Store</summary>

GitHub API | ok
-- | --
GitHub Content | ok
GitHub Web | ok
HACS Data | ok
GitHub API Calls Remaining | 5000
Installed Version | 2.0.5
Stage | running
Available Repositories | 2454
Downloaded Repositories | 3

</details>

<details><summary>Home Assistant Cloud</summary>

logged_in | false
-- | --
can_reach_cert_server | ok
can_reach_cloud_auth | ok
can_reach_cloud | ok

</details>

<details><summary>Home Assistant Supervisor</summary>

host_os | Home Assistant OS 16.3
-- | --
update_channel | stable
supervisor_version | supervisor-2025.11.5
agent_version | 1.7.2
docker_version | 28.3.3
disk_total | 93.8 GB
disk_used | 18.8 GB
nameservers | fd00::ca0e:14ff:fed8:5ab8, 192.168.25.1, 2003:e4:6f4b:9800:ca0e:14ff:fed8:5ab8, 2003:e4:6f42:1200:ca0e:14ff:fed8:5ab8
healthy | true
supported | true
host_connectivity | true
supervisor_connectivity | true
ntp_synchronized | true
virtualization | oracle
board | ova
supervisor_api | ok
version_api | ok
installed_addons | Studio Code Server (6.0.1), InfluxDB (5.0.2), Grafana (12.1.0), File editor (5.8.0), Mosquitto broker (6.5.2), OpenCCU (3.85.7.20251129), SQLite Web (5.0.0), SMA Energy Meter (1.0.0), ESPHome Device Builder (beta) (2025.11.2), Advanced SSH & Web Terminal (22.0.3)

</details>

<details><summary>Dashboards</summary>

dashboards | 4
-- | --
resources | 0
views | 4
mode | storage

</details>

<details><summary>Network Configuration</summary>

adapters | lo (disabled), enp0s3 (enabled, default, auto), docker0 (disabled), hassio (disabled), veth985c031 (disabled), vethc2b2f94 (disabled), veth83ee606 (disabled), veth2889b2a (disabled), veth4210b00 (disabled), vethe2ce969 (disabled), veth696af8e (disabled), veth34ec814 (disabled), veth1f03cce (disabled), vethdb8cfb5 (disabled), veth7974d60 (disabled), veth585a561 (disabled)
-- | --
ipv4_addresses | lo (127.0.0.1/8), enp0s3 (192.168.25.54/24), docker0 (172.30.232.1/23), hassio (172.30.32.1/23), veth985c031 (), vethc2b2f94 (), veth83ee606 (), veth2889b2a (), veth4210b00 (), vethe2ce969 (), veth696af8e (), veth34ec814 (), veth1f03cce (), vethdb8cfb5 (), veth7974d60 (), veth585a561 ()
ipv6_addresses | lo (::1/128), enp0s3 (2003:e4:6f42:1200:3f46:74c3:a7e2:842c/64, fd00::9c76:5427:dfae:262a/64, 2003:e4:6f4b:9800:2c8b:cea4:6280:9944/64, fe80::9fb3:beef:8898:6568/64), docker0 (fe80::bcec:eeff:fed4:88ca/64), hassio (fd0c:ac1e:2100::1/48, fe80::183a:a1ff:fead:65eb/64), veth985c031 (fe80::4426:fbff:fe77:d682/64), vethc2b2f94 (fe80::fcf0:b3ff:fe5e:19cd/64), veth83ee606 (fe80::6018:efff:fe5e:dafd/64), veth2889b2a (fe80::2c32:aeff:fefb:ff0e/64), veth4210b00 (fe80::881:b3ff:fe81:fef1/64), vethe2ce969 (fe80::8077:cdff:fe8c:57d6/64), veth696af8e (fe80::d84e:63ff:fe68:aa9d/64), veth34ec814 (fe80::e800:2bff:fe4d:205d/64), veth1f03cce (fe80::24e2:66ff:fe86:1dc9/64), vethdb8cfb5 (fe80::425:45ff:fe82:c6e/64), veth7974d60 (fe80::407f:b3ff:fe9f:a10e/64), veth585a561 (fe80::98c3:9fff:fe1d:ef3b/64)
announce_addresses | 192.168.25.54, 2003:e4:6f42:1200:3f46:74c3:a7e2:842c, fd00::9c76:5427:dfae:262a, 2003:e4:6f4b:9800:2c8b:cea4:6280:9944, fe80::9fb3:beef:8898:6568

</details>

<details><summary>Recorder</summary>

oldest_recorder_run | 19. November 2025 um 12:06
-- | --
current_recorder_run | 1. Dezember 2025 um 04:12
estimated_db_size | 1085.60 MiB
database_engine | sqlite
database_version | 3.49.2

</details>

Additional information

No response

tzdisco4 avatar Dec 01 '25 10:12 tzdisco4

After downgrading to 16.2 the following issue occoured during boot in the system journal:

Dec 01 11:59:35 homeassistant homeassistant[594]: s6-rc: info: service legacy-services successfully started
Dec 01 11:59:38 homeassistant kernel: BUG: unable to handle page fault for address: ffffffff00000054
Dec 01 11:59:38 homeassistant kernel: #PF: supervisor read access in kernel mode
Dec 01 11:59:38 homeassistant kernel: #PF: error_code(0x0000) - not-present page
Dec 01 11:59:38 homeassistant kernel: PGD 11603d067 P4D 11603d067 PUD 0 
Dec 01 11:59:38 homeassistant kernel: Oops: Oops: 0000 [#1] PREEMPT SMP PTI
Dec 01 11:59:38 homeassistant kernel: CPU: 0 UID: 0 PID: 50 Comm: kswapd0 Tainted: G           O       6.12.43-haos #1
Dec 01 11:59:38 homeassistant kernel: Tainted: [O]=OOT_MODULE
Dec 01 11:59:38 homeassistant kernel: Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Dec 01 11:59:38 homeassistant kernel: RIP: 0010:__remove_mapping+0x9e/0x220
Dec 01 11:59:38 homeassistant kernel: Code: b1 4b 34 39 d0 0f 85 44 01 00 00 48 8b 03 a8 10 0f 85 36 01 00 00 48 8b 03 a9 00 00 02 00 0f 85 98 00 00 00 48 8b 45 68 31 f6 <4c> 8b 78 50 45 84 e4 74 10 48 8b 03 31 f6 a9 00 00 02 00 0f 84 c9
Dec 01 11:59:38 homeassistant kernel: RSP: 0000:ffffb39a001c78b8 EFLAGS: 00010046
Dec 01 11:59:38 homeassistant kernel: RAX: ffffffff00000004 RBX: fffffa47c1138100 RCX: 0000000000000000
Dec 01 11:59:38 homeassistant kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: ffff9935c75a2fa0
Dec 01 11:59:38 homeassistant kernel: RBP: ffff9935c75a2f98 R08: ffffb39a001c7850 R09: 0000000000000000
Dec 01 11:59:38 homeassistant kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
Dec 01 11:59:38 homeassistant kernel: R13: 0000000000000000 R14: ffff9935c75a2fa0 R15: ffffb39a001c7da0
Dec 01 11:59:38 homeassistant kernel: FS:  0000000000000000(0000) GS:ffff9936dbc00000(0000) knlGS:0000000000000000
Dec 01 11:59:38 homeassistant kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 01 11:59:38 homeassistant kernel: CR2: ffffffff00000054 CR3: 0000000003e1e004 CR4: 00000000000706f0
Dec 01 11:59:38 homeassistant kernel: Call Trace:
Dec 01 11:59:38 homeassistant kernel:  <TASK>
Dec 01 11:59:38 homeassistant kernel:  shrink_folio_list+0xa67/0xc40
Dec 01 11:59:38 homeassistant kernel:  evict_folios+0x25e/0x5e0
Dec 01 11:59:38 homeassistant kernel:  try_to_shrink_lruvec+0x19f/0x280
Dec 01 11:59:38 homeassistant kernel:  shrink_one+0xf8/0x1e0
Dec 01 11:59:38 homeassistant kernel:  shrink_node+0xa90/0xc60
Dec 01 11:59:38 homeassistant kernel:  balance_pgdat+0x4aa/0x8e0
Dec 01 11:59:38 homeassistant kernel:  ? update_load_avg+0x77/0x760
Dec 01 11:59:38 homeassistant kernel:  ? __dequeue_entity+0x3d8/0x490
Dec 01 11:59:38 homeassistant kernel:  kswapd+0x1e7/0x390
Dec 01 11:59:38 homeassistant kernel:  ? __pfx_autoremove_wake_function+0x10/0x10
Dec 01 11:59:38 homeassistant kernel:  ? __pfx_kswapd+0x10/0x10
Dec 01 11:59:38 homeassistant kernel:  kthread+0xcb/0xf0
Dec 01 11:59:38 homeassistant kernel:  ? __pfx_kthread+0x10/0x10
Dec 01 11:59:38 homeassistant kernel:  ret_from_fork+0x2f/0x50
Dec 01 11:59:38 homeassistant kernel:  ? __pfx_kthread+0x10/0x10
Dec 01 11:59:38 homeassistant kernel:  ret_from_fork_asm+0x1a/0x30
Dec 01 11:59:38 homeassistant kernel:  </TASK>
Dec 01 11:59:38 homeassistant kernel: Modules linked in: eq3_char_loop(O) rfcomm xt_set ip_set nft_chain_nat nft_compat nf_tables xfrm_user algif_hash algif_skcipher af_alg bnep cfg80211 btusb btmtk btrtl btbcm btintel snd_hda_codec_generic bluetooth snd_hda_intel snd_intel_dspcfg hb_rf_usb_2(O) snd_hda_codec ecdh_generic snd_hwdep sch_fq_codel ecc generic_raw_uart(O) snd_hda_core vboxguest dmi_sysfs
Dec 01 11:59:38 homeassistant kernel: CR2: ffffffff00000054
Dec 01 11:59:38 homeassistant kernel: ---[ end trace 0000000000000000 ]---
Dec 01 11:59:38 homeassistant kernel: RIP: 0010:__remove_mapping+0x9e/0x220
Dec 01 11:59:38 homeassistant kernel: Code: b1 4b 34 39 d0 0f 85 44 01 00 00 48 8b 03 a8 10 0f 85 36 01 00 00 48 8b 03 a9 00 00 02 00 0f 85 98 00 00 00 48 8b 45 68 31 f6 <4c> 8b 78 50 45 84 e4 74 10 48 8b 03 31 f6 a9 00 00 02 00 0f 84 c9
Dec 01 11:59:38 homeassistant kernel: RSP: 0000:ffffb39a001c78b8 EFLAGS: 00010046
Dec 01 11:59:38 homeassistant kernel: RAX: ffffffff00000004 RBX: fffffa47c1138100 RCX: 0000000000000000
Dec 01 11:59:38 homeassistant kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: ffff9935c75a2fa0
Dec 01 11:59:38 homeassistant kernel: RBP: ffff9935c75a2f98 R08: ffffb39a001c7850 R09: 0000000000000000
Dec 01 11:59:38 homeassistant kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
Dec 01 11:59:38 homeassistant kernel: R13: 0000000000000000 R14: ffff9935c75a2fa0 R15: ffffb39a001c7da0
Dec 01 11:59:38 homeassistant kernel: FS:  0000000000000000(0000) GS:ffff9936dbc00000(0000) knlGS:0000000000000000
Dec 01 11:59:38 homeassistant kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 01 11:59:38 homeassistant kernel: CR2: ffffffff00000054 CR3: 0000000003e1e004 CR4: 00000000000706f0
Dec 01 11:59:38 homeassistant kernel: note: kswapd0[50] exited with irqs disabled
Dec 01 11:59:38 homeassistant kernel: note: kswapd0[50] exited with preempt_count 2
Dec 01 11:59:38 homeassistant kernel: ------------[ cut here ]------------
Dec 01 11:59:38 homeassistant kernel: WARNING: CPU: 0 PID: 50 at kernel/exit.c:886 do_exit+0x845/0xa60
Dec 01 11:59:38 homeassistant kernel: Modules linked in: eq3_char_loop(O) rfcomm xt_set ip_set nft_chain_nat nft_compat nf_tables xfrm_user algif_hash algif_skcipher af_alg bnep cfg80211 btusb btmtk btrtl btbcm btintel snd_hda_codec_generic bluetooth snd_hda_intel snd_intel_dspcfg hb_rf_usb_2(O) snd_hda_codec ecdh_generic snd_hwdep sch_fq_codel ecc generic_raw_uart(O) snd_hda_core vboxguest dmi_sysfs
Dec 01 11:59:38 homeassistant kernel: CPU: 0 UID: 0 PID: 50 Comm: kswapd0 Tainted: G      D    O       6.12.43-haos #1
Dec 01 11:59:38 homeassistant kernel: Tainted: [D]=DIE, [O]=OOT_MODULE
Dec 01 11:59:38 homeassistant kernel: Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Dec 01 11:59:38 homeassistant kernel: RIP: 0010:do_exit+0x845/0xa60
Dec 01 11:59:38 homeassistant kernel: Code: 00 00 31 f6 e8 3c e4 ff ff e9 fa fd ff ff 4c 89 e6 bf 05 06 00 00 e8 5a 0c 01 00 e9 ca f8 ff ff 90 0f 0b 90 e9 f8 f7 ff ff 90 <0f> 0b 90 e9 2b f8 ff ff 48 89 df e8 4b 99 10 00 e9 9f f9 ff ff 48
Dec 01 11:59:38 homeassistant kernel: RSP: 0018:ffffb39a001c7ed8 EFLAGS: 00010286
Dec 01 11:59:38 homeassistant kernel: RAX: 0000000000000001 RBX: ffff9936c0a0a000 RCX: 0000000000000000
Dec 01 11:59:38 homeassistant kernel: RDX: 0000000000000001 RSI: 0000000000002710 RDI: ffff9936c082a940
Dec 01 11:59:38 homeassistant kernel: RBP: ffff9936c03f5540 R08: 00000000ffffdfff R09: 0000000000000001
Dec 01 11:59:38 homeassistant kernel: R10: 00000000ffffdfff R11: ffffffffab465220 R12: 0000000000000009
Dec 01 11:59:38 homeassistant kernel: R13: ffff9936c082a940 R14: 0000000000000000 R15: 0000000000000000
Dec 01 11:59:38 homeassistant kernel: FS:  0000000000000000(0000) GS:ffff9936dbc00000(0000) knlGS:0000000000000000
Dec 01 11:59:38 homeassistant kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 01 11:59:38 homeassistant kernel: CR2: ffffffff00000054 CR3: 0000000108e34003 CR4: 00000000000706f0
Dec 01 11:59:38 homeassistant kernel: Call Trace:
Dec 01 11:59:38 homeassistant kernel:  <TASK>
Dec 01 11:59:38 homeassistant kernel:  make_task_dead+0x7c/0x160
Dec 01 11:59:38 homeassistant kernel:  rewind_stack_and_make_dead+0x16/0x20
Dec 01 11:59:38 homeassistant kernel: RIP: 0000:0x0
Dec 01 11:59:38 homeassistant kernel: Code: Unable to access opcode bytes at 0xffffffffffffffd6.
Dec 01 11:59:38 homeassistant kernel: RSP: 0000:0000000000000000 EFLAGS: 00000000 ORIG_RAX: 0000000000000000
Dec 01 11:59:38 homeassistant kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec 01 11:59:38 homeassistant kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec 01 11:59:38 homeassistant kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Dec 01 11:59:38 homeassistant kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec 01 11:59:38 homeassistant kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec 01 11:59:38 homeassistant kernel:  </TASK>
Dec 01 11:59:38 homeassistant kernel: ---[ end trace 0000000000000000 ]---
Dec 01 11:59:40 homeassistant homeassistant[594]: 2025-12-01 11:59:40.872 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration

I attach the complete journal.

journal-2.tar.gz

tzdisco4 avatar Dec 01 '25 11:12 tzdisco4

I just checked the systemd journal for segfaults which might have happened since the downgrade yesterday. During an uptime of ~22 hours up to now "zero" segfaults. As 16.2 is running on the same vbox like the 16.3 before it seems obvious, that the HA OS package 16.3 is buggy. I will continue to monitor this.

tzdisco4 avatar Dec 02 '25 08:12 tzdisco4

Current uptime is: 10:03:33 up 1 day, 22:05, 0 users, load average: 0.53, 0.35, 0.25 Still no segfaults or other issues (except the DNS message spamming https://github.com/home-assistant/core/issues/150563).

tzdisco4 avatar Dec 03 '25 09:12 tzdisco4

Current uptime is: 11:49:36 up 2 days, 23:51, 0 users, load average: 0.05, 0.11, 0.11 Still no segfaults or other issues. I'll stop reporting now. To me it seems that the instability of 16.3 is caused by software, not hardware.

tzdisco4 avatar Dec 04 '25 10:12 tzdisco4