core ZHA fails to load with `KeyError: 4` after updating to 2023.2

The problem

ZHA stopped working after upgrading from 2023.1.6 to 2023.2.0. Restoring the backup I made prior to upgrading fixed things. I've also tried upgrading to 2023.2.1, 2023.2.2 and 2023.2.3 with the same thing happening. Restoring the backup causes ZHA to work again. Might be related to: https://github.com/home-assistant/core/issues/87275

This is what my logs say:

`Logger: homeassistant.config_entries Source: components/zha/core/gateway.py:174 First occurred: 14:25:52 (1 occurrences) Last logged: 14:25:52

Error setting up entry USB Serial, s/n: n/a for zha Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/config_entries.py", line 382, in async_setup result = await component.async_setup_entry(hass, self) File "/usr/src/homeassistant/homeassistant/components/zha/init.py", line 111, in async_setup_entry await zha_gateway.async_initialize() File "/usr/src/homeassistant/homeassistant/components/zha/core/gateway.py", line 189, in async_initialize raise exc File "/usr/src/homeassistant/homeassistant/components/zha/core/gateway.py", line 174, in async_initialize self.application_controller = await app_controller_cls.new( File "/usr/local/lib/python3.10/site-packages/zigpy/application.py", line 163, in new await app._load_db() File "/usr/local/lib/python3.10/site-packages/zigpy/application.py", line 80, in _load_db await self._dblistener.load() File "/usr/local/lib/python3.10/site-packages/zigpy/appdb.py", line 581, in load await self._load_unsupported_attributes() File "/usr/local/lib/python3.10/site-packages/zigpy/appdb.py", line 634, in _load_unsupported_attributes ep = dev.endpoints[endpoint_id] KeyError: 4`

`Logger: homeassistant.components.zha.core.gateway Source: components/zha/core/gateway.py:174 Integration: Zigbee Home Automation (documentation, issues) First occurred: 14:25:45 (3 occurrences) Last logged: 14:25:52

Couldn't start ZNP = Texas Instruments Z-Stack ZNP protocol: CC253x, CC26x2, CC13x2 coordinator (attempt 1 of 3) Couldn't start ZNP = Texas Instruments Z-Stack ZNP protocol: CC253x, CC26x2, CC13x2 coordinator (attempt 2 of 3) Couldn't start ZNP = Texas Instruments Z-Stack ZNP protocol: CC253x, CC26x2, CC13x2 coordinator (attempt 3 of 3) Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/zha/core/gateway.py", line 174, in async_initialize self.application_controller = await app_controller_cls.new( File "/usr/local/lib/python3.10/site-packages/zigpy/application.py", line 163, in new await app._load_db() File "/usr/local/lib/python3.10/site-packages/zigpy/application.py", line 80, in _load_db await self._dblistener.load() File "/usr/local/lib/python3.10/site-packages/zigpy/appdb.py", line 581, in load await self._load_unsupported_attributes() File "/usr/local/lib/python3.10/site-packages/zigpy/appdb.py", line 634, in _load_unsupported_attributes ep = dev.endpoints[endpoint_id] KeyError: 4`

What version of Home Assistant Core has the issue?

2023.2.x

What was the last working version of Home Assistant Core?

2023.1.6

What type of installation are you running?

Home Assistant OS

Integration causing the issue

ZHA

Link to integration documentation on our website

https://www.home-assistant.io/integrations/zha/

Diagnostics information

config_entry-zha-03f8217f51052c34c2126ba5d12f0fc2.json.txt home-assistant_2023-02-03T13-26-17.898Z.log home-assistant_2023-02-03T13-19-58.009Z.log

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

Feb 08 '23 15:02 rjkruit

Hey there @dmulcahey, @adminiuga, @puddly, mind taking a look at this issue as it has been labeled with an integration (zha) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of zha can trigger bot actions by commenting:

@home-assistant close Closes the issue.
@home-assistant rename Awesome new title Change the title of the issue.
@home-assistant reopen Reopen the issue.
@home-assistant unassign zha Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

zha documentation zha source _{^{(message by IssueLinks)}}

Feb 08 '23 15:02 home-assistant[bot]

I have the same errors:

Logger: aiohttp.server Source: components/zha/radio_manager.py:137 First occurred: 11:54:27 AM (1 occurrences) Last logged: 11:54:27 AM

Error handling request Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/serial/serialposix.py", line 322, in open self.fd = os.open(self.portstr, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK) FileNotFoundError: [Errno 2] No such file or directory: '/dev/serial/by-id/usb-Nabu_Casa_SkyConnect_v1.0_40ab4b73b614ed11b20fb68be054580b-if00-port0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request resp = await request_handler(request) File "/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py", line 504, in _handle resp = await handler(request) File "/usr/local/lib/python3.10/site-packages/aiohttp/web_middlewares.py", line 117, in impl return await handler(request) File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 60, in security_filter_middleware return await handler(request) File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 100, in forwarded_middleware return await handler(request) File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware return await handler(request) File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 80, in ban_middleware return await handler(request) File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 236, in auth_middleware return await handler(request) File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 145, in handle result = await result File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 241, in post return await super().post(request, flow_id) File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 72, in wrapper result = await method(view, request, data, *args, **kwargs) File "/usr/src/homeassistant/homeassistant/helpers/data_entry_flow.py", line 110, in post result = await self._flow_mgr.async_configure(flow_id, data) File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 253, in async_configure result = await self._async_handle_step( File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 335, in _async_handle_step result: FlowResult = await getattr(flow, method)(user_input) File "/usr/src/homeassistant/homeassistant/components/zha/config_flow.py", line 642, in async_step_intent_migrate await self._radio_mgr.async_reset_adapter() File "/usr/src/homeassistant/homeassistant/components/zha/radio_manager.py", line 221, in async_reset_adapter async with self._connect_zigpy_app() as app: File "/usr/local/lib/python3.10/contextlib.py", line 199, in aenter return await anext(self.gen) File "/usr/src/homeassistant/homeassistant/components/zha/radio_manager.py", line 137, in _connect_zigpy_app await app.connect() File "/usr/local/lib/python3.10/site-packages/bellows/zigbee/application.py", line 124, in connect self._ezsp = await bellows.ezsp.EZSP.initialize(self.config) File "/usr/local/lib/python3.10/site-packages/bellows/ezsp/init.py", line 101, in initialize await ezsp.connect(use_thread=zigpy_config[conf.CONF_USE_THREAD]) File "/usr/local/lib/python3.10/site-packages/bellows/ezsp/init.py", line 118, in connect self._gw = await bellows.uart.connect(self._config, self, use_thread=use_thread) File "/usr/local/lib/python3.10/site-packages/bellows/uart.py", line 396, in connect protocol, connection_done = await thread.run_coroutine_threadsafe( File "/usr/local/lib/python3.10/site-packages/bellows/uart.py", line 375, in _connect transport, protocol = await zigpy.serial.create_serial_connection( File "/usr/local/lib/python3.10/site-packages/zigpy/serial.py", line 40, in create_serial_connection transport, protocol = await pyserial_asyncio.create_serial_connection( File "/usr/local/lib/python3.10/site-packages/serial_asyncio/init.py", line 448, in create_serial_connection serial_instance = serial.serial_for_url(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/serial/init.py", line 90, in serial_for_url instance.open() File "/usr/local/lib/python3.10/site-packages/serial/serialposix.py", line 325, in open raise SerialException(msg.errno, "could not open port {}: {}".format(self._port, msg)) serial.serialutil.SerialException: [Errno 2] could not open port /dev/serial/by-id/usb-Nabu_Casa_SkyConnect_v1.0_40ab4b73b614ed11b20fb68be054580b-if00-port0: [Errno 2] No such file or directory: '/dev/serial/by-id/usb-Nabu_Casa_SkyConnect_v1.0_40ab4b73b614ed11b20fb68be054580b-if00-port0'

Logger: homeassistant.components.zha.core.gateway Source: components/zha/core/gateway.py:174 Integration: Zigbee Home Automation (documentation, issues) First occurred: 8:17:49 AM (9 occurrences) Last logged: 11:54:35 AM

Couldn't start EZSP = Silicon Labs EmberZNet protocol: Elelabs, HUSBZB-1, Telegesis coordinator (attempt 1 of 3) Couldn't start EZSP = Silicon Labs EmberZNet protocol: Elelabs, HUSBZB-1, Telegesis coordinator (attempt 2 of 3) Couldn't start EZSP = Silicon Labs EmberZNet protocol: Elelabs, HUSBZB-1, Telegesis coordinator (attempt 3 of 3) Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/serial/serialposix.py", line 322, in open self.fd = os.open(self.portstr, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK) FileNotFoundError: [Errno 2] No such file or directory: '/dev/serial/by-id/usb-Nabu_Casa_SkyConnect_v1.0_40ab4b73b614ed11b20fb68be054580b-if00-port0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/zha/core/gateway.py", line 174, in async_initialize self.application_controller = await app_controller_cls.new( File "/usr/local/lib/python3.10/site-packages/zigpy/application.py", line 169, in new await app.startup(auto_form=auto_form) File "/usr/local/lib/python3.10/site-packages/zigpy/application.py", line 145, in startup await self.connect() File "/usr/local/lib/python3.10/site-packages/bellows/zigbee/application.py", line 124, in connect self._ezsp = await bellows.ezsp.EZSP.initialize(self.config) File "/usr/local/lib/python3.10/site-packages/bellows/ezsp/init.py", line 101, in initialize await ezsp.connect(use_thread=zigpy_config[conf.CONF_USE_THREAD]) File "/usr/local/lib/python3.10/site-packages/bellows/ezsp/init.py", line 118, in connect self._gw = await bellows.uart.connect(self._config, self, use_thread=use_thread) File "/usr/local/lib/python3.10/site-packages/bellows/uart.py", line 396, in connect protocol, connection_done = await thread.run_coroutine_threadsafe( File "/usr/local/lib/python3.10/site-packages/bellows/uart.py", line 375, in _connect transport, protocol = await zigpy.serial.create_serial_connection( File "/usr/local/lib/python3.10/site-packages/zigpy/serial.py", line 40, in create_serial_connection transport, protocol = await pyserial_asyncio.create_serial_connection( File "/usr/local/lib/python3.10/site-packages/serial_asyncio/init.py", line 448, in create_serial_connection serial_instance = serial.serial_for_url(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/serial/init.py", line 90, in serial_for_url instance.open() File "/usr/local/lib/python3.10/site-packages/serial/serialposix.py", line 325, in open raise SerialException(msg.errno, "could not open port {}: {}".format(self._port, msg)) serial.serialutil.SerialException: [Errno 2] could not open port /dev/serial/by-id/usb-Nabu_Casa_SkyConnect_v1.0_40ab4b73b614ed11b20fb68be054580b-if00-port0: [Errno 2] No such file or directory: '/dev/serial/by-id/usb-Nabu_Casa_SkyConnect_v1.0_40ab4b73b614ed11b20fb68be054580b-if00-port0'

Logger: homeassistant.config_entries Source: components/zha/core/gateway.py:174 First occurred: 8:17:55 AM (3 occurrences) Last logged: 11:54:35 AM

Error setting up entry SkyConnect v1.0 for zha Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/serial/serialposix.py", line 322, in open self.fd = os.open(self.portstr, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK) FileNotFoundError: [Errno 2] No such file or directory: '/dev/serial/by-id/usb-Nabu_Casa_SkyConnect_v1.0_40ab4b73b614ed11b20fb68be054580b-if00-port0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/config_entries.py", line 382, in async_setup result = await component.async_setup_entry(hass, self) File "/usr/src/homeassistant/homeassistant/components/zha/init.py", line 111, in async_setup_entry await zha_gateway.async_initialize() File "/usr/src/homeassistant/homeassistant/components/zha/core/gateway.py", line 189, in async_initialize raise exc File "/usr/src/homeassistant/homeassistant/components/zha/core/gateway.py", line 174, in async_initialize self.application_controller = await app_controller_cls.new( File "/usr/local/lib/python3.10/site-packages/zigpy/application.py", line 169, in new await app.startup(auto_form=auto_form) File "/usr/local/lib/python3.10/site-packages/zigpy/application.py", line 145, in startup await self.connect() File "/usr/local/lib/python3.10/site-packages/bellows/zigbee/application.py", line 124, in connect self._ezsp = await bellows.ezsp.EZSP.initialize(self.config) File "/usr/local/lib/python3.10/site-packages/bellows/ezsp/init.py", line 101, in initialize await ezsp.connect(use_thread=zigpy_config[conf.CONF_USE_THREAD]) File "/usr/local/lib/python3.10/site-packages/bellows/ezsp/init.py", line 118, in connect self._gw = await bellows.uart.connect(self._config, self, use_thread=use_thread) File "/usr/local/lib/python3.10/site-packages/bellows/uart.py", line 396, in connect protocol, connection_done = await thread.run_coroutine_threadsafe( File "/usr/local/lib/python3.10/site-packages/bellows/uart.py", line 375, in _connect transport, protocol = await zigpy.serial.create_serial_connection( File "/usr/local/lib/python3.10/site-packages/zigpy/serial.py", line 40, in create_serial_connection transport, protocol = await pyserial_asyncio.create_serial_connection( File "/usr/local/lib/python3.10/site-packages/serial_asyncio/init.py", line 448, in create_serial_connection serial_instance = serial.serial_for_url(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/serial/init.py", line 90, in serial_for_url instance.open() File "/usr/local/lib/python3.10/site-packages/serial/serialposix.py", line 325, in open raise SerialException(msg.errno, "could not open port {}: {}".format(self._port, msg)) serial.serialutil.SerialException: [Errno 2] could not open port /dev/serial/by-id/usb-Nabu_Casa_SkyConnect_v1.0_40ab4b73b614ed11b20fb68be054580b-if00-port0: [Errno 2] No such file or directory: '/dev/serial/by-id/usb-Nabu_Casa_SkyConnect_v1.0_40ab4b73b614ed11b20fb68be054580b-if00-port0'

Feb 08 '23 17:02 samsonusmc

@samsonusmc Please open a new issue, your problem is unrelated to this one.

Feb 08 '23 17:02 puddly

I need to know how you think it's different so I can dive into it and provide details. it's the same error with the same problem II just provided additional logs, the middle section is the same as the OP. Broke with 2023.2 and works with 2023.1?

logger: homeassistant.components.zha.core.gateway Source: components/zha/core/gateway.py:174 Integration: Zigbee Home Automation (documentation, issues) First occurred: 8:17:49 AM (9 occurrences) Last logged: 11:54:35 AM

Feb 08 '23 17:02 samsonusmc

@samsonusmc Your error is FileNotFoundError: [Errno 2] No such file or directory: '/dev/serial/by-id/usb-Nabu_Casa_SkyConnect_v1.0_40ab4b73b614ed11b20fb68be054580b-if00-port0', the current issue's error is KeyError: 4.

They have similar side effects (i.e. ZHA doesn't load), but are otherwise completely unrelated and likely do not have any shared root cause. Please open a new issue and fill out the issue template and include what installation method you are using and if a VM running HA OS, what type of VM.

Feb 08 '23 17:02 puddly

I am also having the same issue as the original poster - keyerror: 4

When I did the update to 2023.2 I was using the Sonoff dongle P and was getting this error. I held off doing anything in the (false) hope that the new SkyConnect would fix things, it did not.

I am running HA in a VMware VM on a Windows box. OS Version 9.5 & now Core 2023.2.3

Logger: homeassistant.config_entries Source: components/zha/core/gateway.py:174 First occurred: 6:38:27 PM (1 occurrences) Last logged: 6:38:27 PM

Error setting up entry SkyConnect v1.0 for zha Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/config_entries.py", line 382, in async_setup result = await component.async_setup_entry(hass, self) File "/usr/src/homeassistant/homeassistant/components/zha/init.py", line 111, in async_setup_entry await zha_gateway.async_initialize() File "/usr/src/homeassistant/homeassistant/components/zha/core/gateway.py", line 189, in async_initialize raise exc File "/usr/src/homeassistant/homeassistant/components/zha/core/gateway.py", line 174, in async_initialize self.application_controller = await app_controller_cls.new( File "/usr/local/lib/python3.10/site-packages/zigpy/application.py", line 163, in new await app._load_db() File "/usr/local/lib/python3.10/site-packages/zigpy/application.py", line 80, in _load_db await self._dblistener.load() File "/usr/local/lib/python3.10/site-packages/zigpy/appdb.py", line 581, in load await self._load_unsupported_attributes() File "/usr/local/lib/python3.10/site-packages/zigpy/appdb.py", line 634, in _load_unsupported_attributes ep = dev.endpoints[endpoint_id] KeyError: 4

Feb 10 '23 23:02 jporterny

@jporterny did you have any luck with this? I couldn’t find any settings in ZHA that might fix this and I have no idea where else to look.

Feb 12 '23 22:02 rjkruit

@jporterny did you have any luck with this? I couldn’t find any settings in ZHA that might fix this and I have no idea where else to look.

The only thing I have been able to do is make a new VM and restore from a backup to before the 2023-2. I have not heard anything else on this issue.

Feb 18 '23 02:02 jporterny

So I've just updated to 2023.2.5 since the release notes listed that the matter service was updated and guess what? ZHA works!

Feb 19 '23 21:02 rjkruit

I gave up waiting and made a new install in a new VM. I was able to install every thing including ZHA with no issues. I also use the SkyConnect stick instead of the one I had been using previously.

I am no longer in need of a fix or resolution for this issue.

Mar 18 '23 23:03 jporterny

A fix for this issue has been merged (https://github.com/zigpy/zigpy/pull/1181) and will be included in a future Home Assistant bugfix release.

Mar 18 '23 23:03 puddly

The fix for this is in the current beta.

Mar 31 '23 19:03 puddly