supervisor icon indicating copy to clipboard operation
supervisor copied to clipboard

Supervisor forgets the default interface's static network addresses after router restart (but not always)

Open lmagyar opened this issue 1 year ago • 16 comments

Describe the issue you are experiencing

After router restart, Supervisor API returns empty network interface address array. The IPv4 address is static, no DHCP requested, though the router has DHCP server. ha su reload doesn't help ha su restart needed to get valid values back And it doesn't happen all the time, "just" approx. 50% of the time. Any add-on that uses these information, will malfunction.

~ # bashio::api.supervisor GET "/network/interface/default/info" false | jq -M
{
  "interface": "enu1u1",
  "type": "ethernet",
  "enabled": true,
  "connected": true,
  "primary": true,
  "mac": "B8:27:EB:2F:8F:FA",
  "ipv4": {
    "method": "static",
    "address": [],    <<<<<<<<<<<<<<<<<<<<<<<<<<< ?!?!?!?!?!?!
    "nameservers": [],
    "gateway": null,
    "ready": true
  },
  "ipv6": {
    "method": "disabled",
    "address": [],
    "nameservers": [],
    "gateway": null,
    "ready": true
  },
  "wifi": null,
  "vlan": null
}

And it has address, I'm accessing these info through a functioning interface...

$ ifconfig enu1u1
enu1u1    Link encap:Ethernet  HWaddr B8:27:EB:2F:8F:FA
          inet addr:192.168.1.8  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::2451:3dcb:3eda:78fa/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:163135 errors:0 dropped:0 overruns:0 frame:0
          TX packets:110955 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:124040140 (118.2 MiB)  TX bytes:15715734 (14.9 MiB)

What type of installation are you running?

Home Assistant OS

Which operating system are you running on?

Home Assistant Operating System

Steps to reproduce the issue

  1. ha net info - OK
  2. reboot router (Mikrotik)
  3. ha net info - missing info

Anything in the Supervisor logs that might be useful for us?

The supervisor log is not so intresting, I've merged the Network Manager's events from one of my add-on, plus the router's events with the Supervisor log.

Note: Router reboot started, router is down

[20:32:59] INFO: Handling Network Manager action enu1u1 down
Note: Supervisor has valid address at this time.

2024-10-18 22:32:59.400 INFO (MainThread) [supervisor.plugins.dns] Restarting CoreDNS plugin
2024-10-18 22:32:59.461 INFO (MainThread) [supervisor.host.network] Updating local network information
2024-10-18 22:33:00.111 INFO (SyncWorker_1) [supervisor.docker.manager] Restarting hassio_dns
2024-10-18 22:33:03.857 INFO (MainThread) [supervisor.api.middleware.security] /network/info access from 5a1729ad_tailscale-beta
2024-10-18 22:33:04.441 INFO (MainThread) [supervisor.api.middleware.security] /network/interface/enu1u1/info access from 5a1729ad_tailscale-beta
2024-10-18 22:33:04.779 INFO (MainThread) [supervisor.api.middleware.security] /network/interface/wlan0/info access from 5a1729ad_tailscale-beta

[20:33:06] INFO: Handling Network Manager action enu1u1 up
Note: Supervisor has valid address at this time.

2024-10-18 22:33:08.081 INFO (MainThread) [supervisor.api.middleware.security] /network/info access from 5a1729ad_tailscale-beta
2024-10-18 22:33:08.302 INFO (MainThread) [supervisor.api.middleware.security] /network/interface/enu1u1/info access from 5a1729ad_tailscale-beta
2024-10-18 22:33:08.589 INFO (MainThread) [supervisor.api.middleware.security] /network/interface/wlan0/info access from 5a1729ad_tailscale-beta



Note: Router rebooted, though not 100% functioning yet

[20:35:38] INFO: Handling Network Manager action enu1u1 down
Note: Addresses are gone now at Supervisor, and never came back...

2024-10-18 22:35:39.356 INFO (MainThread) [supervisor.api.middleware.security] /network/info access from 5a1729ad_tailscale-beta
2024-10-18 22:35:39.582 INFO (MainThread) [supervisor.api.middleware.security] /network/interface/enu1u1/info access from 5a1729ad_tailscale-beta
2024-10-18 22:35:39.865 INFO (MainThread) [supervisor.api.middleware.security] /network/interface/wlan0/info access from 5a1729ad_tailscale-beta

[20:35:41] INFO: Handling Network Manager action enu1u1 up
Note: Addresses are still gone...

2024-10-18 22:35:41.154 INFO (MainThread) [supervisor.plugins.dns] Restarting CoreDNS plugin
2024-10-18 22:35:41.725 INFO (SyncWorker_6) [supervisor.docker.manager] Restarting hassio_dns
2024-10-18 22:35:43.275 INFO (MainThread) [supervisor.api.middleware.security] /network/info access from 5a1729ad_tailscale-beta
2024-10-18 22:35:43.511 INFO (MainThread) [supervisor.api.middleware.security] /network/interface/enu1u1/info access from 5a1729ad_tailscale-beta
2024-10-18 22:35:43.780 INFO (MainThread) [supervisor.api.middleware.security] /network/interface/wlan0/info access from 5a1729ad_tailscale-beta



Note: Router is completely up and functional

[20:36:10] INFO: Handling Network Manager action enu1u1 down
Note: Addresses are still gone...

2024-10-18 22:36:11.967 INFO (MainThread) [supervisor.api.middleware.security] /network/info access from 5a1729ad_tailscale-beta
2024-10-18 22:36:12.212 INFO (MainThread) [supervisor.api.middleware.security] /network/interface/enu1u1/info access from 5a1729ad_tailscale-beta
2024-10-18 22:36:12.503 INFO (MainThread) [supervisor.api.middleware.security] /network/interface/wlan0/info access from 5a1729ad_tailscale-beta

[20:36:15] INFO: Handling Network Manager action enu1u1 up
Note: Addresses are still gone!!!!!!!!!!!!!!!

2024-10-18 22:36:15.218 INFO (MainThread) [supervisor.plugins.dns] Restarting CoreDNS plugin
2024-10-18 22:36:15.735 INFO (SyncWorker_0) [supervisor.docker.manager] Restarting hassio_dns
2024-10-18 22:36:18.401 INFO (MainThread) [supervisor.api.middleware.security] /network/info access from 5a1729ad_tailscale-beta
2024-10-18 22:36:18.706 INFO (MainThread) [supervisor.api.middleware.security] /network/interface/enu1u1/info access from 5a1729ad_tailscale-beta
2024-10-18 22:36:19.196 INFO (MainThread) [supervisor.api.middleware.security] /network/interface/wlan0/info access from 5a1729ad_tailscale-beta
2024-10-18 22:36:49.457 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state running
2024-10-18 22:36:49.465 INFO (MainThread) [supervisor.resolution.checks.base] Run check for trust/supervisor
2024-10-18 22:36:49.872 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_ipv6_error/dns_server
2024-10-18 22:36:50.826 INFO (MainThread) [supervisor.resolution.checks.base] Run check for multiple_data_disks/system
2024-10-18 22:36:50.827 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_failed/dns_server
2024-10-18 22:36:50.829 INFO (MainThread) [supervisor.resolution.checks.base] Run check for pwned/addon
2024-10-18 22:36:50.830 INFO (MainThread) [supervisor.resolution.checks.base] Run check for free_space/system
2024-10-18 22:36:50.848 INFO (MainThread) [supervisor.resolution.checks.base] Run check for security/core
2024-10-18 22:36:50.855 INFO (MainThread) [supervisor.resolution.checks.base] Run check for disabled_data_disk/system
2024-10-18 22:36:50.856 INFO (MainThread) [supervisor.resolution.checks.base] Run check for no_current_backup/system
2024-10-18 22:36:50.865 INFO (MainThread) [supervisor.resolution.checks.base] Run check for docker_config/system
2024-10-18 22:36:50.866 INFO (MainThread) [supervisor.resolution.checks.base] Run check for ipv4_connection_problem/system
2024-10-18 22:36:50.868 INFO (MainThread) [supervisor.resolution.check] System checks complete
2024-10-18 22:36:50.869 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state running
2024-10-18 22:36:53.343 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
2024-10-18 22:36:53.344 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state running
2024-10-18 22:36:53.345 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
2024-10-18 22:38:17.074 INFO (MainThread) [supervisor.api.middleware.security] /network/info access from a0d7b954_ssh
2024-10-18 22:38:17.112 INFO (MainThread) [supervisor.api.middleware.security] /network/info access from a0d7b954_ssh
2024-10-18 22:38:17.119 INFO (MainThread) [supervisor.api.middleware.security] /host/info access from a0d7b954_ssh
2024-10-18 22:38:17.145 INFO (MainThread) [supervisor.api.middleware.security] /core/info access from a0d7b954_ssh
2024-10-18 22:38:21.968 INFO (MainThread) [supervisor.api.middleware.security] /network/info access from a0d7b954_ssh
2024-10-18 22:38:37.703 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
...

System Health information

System Information

version core-2024.10.1
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.12.4
os_name Linux
os_version 6.6.31-haos-raspi
arch aarch64
timezone Europe/Budapest
config_dir /config
Home Assistant Community Store
GitHub API ok
GitHub Content ok
GitHub Web ok
HACS Data ok
GitHub API Calls Remaining 4994
Installed Version 2.0.1
Stage running
Available Repositories 1520
Downloaded Repositories 7
Home Assistant Supervisor
host_os Home Assistant OS 13.1
update_channel stable
supervisor_version supervisor-2024.10.2
agent_version 1.6.0
docker_version 26.1.4
disk_total 27.7 GB
disk_used 8.6 GB
healthy true
supported true
host_connectivity true
supervisor_connectivity true
ntp_synchronized true
virtualization
board rpi3-64
supervisor_api ok
version_api ok
installed_addons File editor (5.8.0), Advanced SSH & Web Terminal (19.0.0), [BETA] Tailscale with features (0.22.1.2-rc10), Tasmota proxy and update (0.1.16), [BETA] In-memory MariaDB (2.7.1.3-rc01), [BETA] Portainer (2.4.7-rc01), [BETA] Samba share with features (12.3.2.1-rc01), Duck DNS (1.18.0), NGINX Home Assistant SSL proxy (3.11.0), Mosquitto broker (6.4.1)
Dashboards
dashboards 2
resources 4
views 1
mode storage
Recorder
oldest_recorder_run October 17, 2024 at 22:40
current_recorder_run October 18, 2024 at 00:40
estimated_db_size 6.68 MiB
database_engine mysql
database_version 10.11.8

Supervisor diagnostics

config_entry-hassio-2c83003182a696b1043f39a03bb886ef.json

Additional information

No response

lmagyar avatar Oct 18 '24 21:10 lmagyar

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Nov 18 '24 21:11 github-actions[bot]

I've made some tests:

  • different routers: not relevant
  • different OS-s: relevant! It seems that
    • rpi3-32 nearly always lose the static IP address when router restarted
    • rpi3-64 (original post above) a bit less often
    • rpi4-64 ~never lost the static IP when router restarted~ UPDATE: lost it, but in a really strange way: several UTP cable unplug, plug, router restart, etc. and
      • after a Network Manager action end0 up, SU reported no addresses
      • but a few hours later when I tested, it reported the valid data
      • Note: this is not a bashio caching issue on client side, there is a cache.flush_all every time before I call the SU API

"Lost static IP" means: it works, but ha net info shows empty address lists.

lmagyar avatar Nov 20 '24 11:11 lmagyar

More real life tests, and rpi4-64 also lost the static addresses, previous comment updated.

It seems to me, that this is some kind of timing issue when SU collects these information, and some caching (I mean caching on OS or SU level, not on client of SU API) makes the error permanent on rpi3, but not on rpi4 OS versions. But this is a guess only.

lmagyar avatar Nov 23 '24 13:11 lmagyar

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Dec 23 '24 14:12 github-actions[bot]

not stale

lmagyar avatar Dec 24 '24 10:12 lmagyar

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Jan 23 '25 12:01 github-actions[bot]

not stale, still not fixed, basically any network "change" requires a ha su restart

lmagyar avatar Jan 26 '25 12:01 lmagyar

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Feb 25 '25 13:02 github-actions[bot]

not stale

lmagyar avatar Feb 26 '25 19:02 lmagyar

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Mar 28 '25 20:03 github-actions[bot]

not stale

lmagyar avatar Mar 29 '25 01:03 lmagyar

I've the same issue.. I've different VLAN, I need static IPv4 Andreas on a specific sub-interface but every time after reboot return in automatic (DHCP)

lorenzo-deluca avatar Apr 05 '25 21:04 lorenzo-deluca

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar May 05 '25 22:05 github-actions[bot]

not stale

lmagyar avatar May 05 '25 22:05 lmagyar

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Jun 04 '25 23:06 github-actions[bot]

not stale

lmagyar avatar Jun 06 '25 23:06 lmagyar

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Jul 07 '25 00:07 github-actions[bot]

not stale

lmagyar avatar Jul 07 '25 12:07 lmagyar

Are you sure this is still the case? At least since #5903 we reload the network info on every primary interface change. From your logs it seems that NetworkManager really considers the interface to be down and back up. I would expect that this causes a primary connection change, which we listen for since Supervisor 2025.05.2.

In a quick test by disconnecting/reconnecting the virtual network interface on my Proxmox instance Supervisor every time changed from empty ha network info to the complete set of IP addresses within a few seconds.

If you can still reproduce, please enable the Supervisor debug logs and share the complete log of such a router reboot where Supervisor doesn't refresh the network information (along with the Host logs to see the timing of the NetworkManager events). This should give us all changed NetworkManager properties.

ha supervisor options --logging debug

agners avatar Jul 08 '25 07:07 agners

Thank you for the update, yes, sorry, I didn't retest it when I see no response.

But now I've retested it with rPI3-32 and rPI4-64, SU 2025.6.2, 2025.7.0 and 2025.7.1 Mikrotik router got restarted 10+ times, and I wasn't able to reproduce this issue anymore!

Though I've found a new one https://github.com/hassio-addons/bashio/pull/168 (it's already merged)

lmagyar avatar Jul 09 '25 14:07 lmagyar