Device is unavailable in home assistant after restart, but responsive in Zigbee2mqtt hassio addon gui

[craigwitter]

What happened?

Since the latest update to home assistant ( 2023.11.1) / Z2M, all of my VZM35-SN devices are suddenly unavailable in HomeAssistant via the front end after a restart. The actual fan entity shows "This entity is no longer being provided by the mqtt integration. If the entity is no longer in use, delete it in settings." If I go into the Z2M front end, I can turn the fan on/off without issue. It seems that restarting the Z2M service is the only way to bring back the functionality. When home assistant is still starting up, the items are initially available, but once it goes to wrapping things up, then they all switch to unavailable.

What did you expect to happen?

I expected the devices to be available without having to restart the Z2M addon

How to reproduce it (minimal and precise)

1. Restart home assistant, note that fans show as unavailable in home assistant front end
2. Go into Z2M frontend, verify that they can be controlled
3. Restart Z2M addon, verify that fans show available in home assistant front end and can be controlled

Zigbee2MQTT version

1.33.2-dev

Adapter firmware version

20221226

Adapter

S20221226

Debug log

No errors in the Z2M addon log.

The entry for fan entity in mqtt is missing until the Z2M addon is reloaded. Here is the MQTT info for one random fan device: Master Bedroom Fan (fan.master_bedroom_fan) MQTT discovery data: Topic: homeassistant/fan/0x048727fffe1c75b8/fan/config Payload { "availability": [ { "topic": "zigbee2mqtt/bridge/state", "value_template": "{{ value_json.state }}" }, { "topic": "zigbee2mqtt/Master Bedroom Fan/availability", "value_template": "{{ value_json.state }}" } ], "availability_mode": "all", "command_topic": "zigbee2mqtt/Master Bedroom Fan/set/fan_state", "device": { "identifiers": [ "zigbee2mqtt_0x048727fffe1c75b8" ], "manufacturer": "Inovelli", "model": "Fan controller (VZM35-SN)", "name": "Master Bedroom Fan", "sw_version": "V.04" }, "name": null, "object_id": "master_bedroom_fan", "origin": { "name": "Zigbee2MQTT", "sw_version": "1.33.2-dev", "support_url": "https://www.zigbee2mqtt.io" }, "percentage_command_template": "{{ {0:'off', 1:'low', 2:'medium', 3:'high'}[value] | default('') }}", "percentage_command_topic": "zigbee2mqtt/Master Bedroom Fan/set/fan_mode", "percentage_state_topic": "zigbee2mqtt/Master Bedroom Fan", "percentage_value_template": "{{ {'off':0, 'low':1, 'medium':2, 'high':3}[value_json.fan_mode] | default('None') }}", "preset_mode_command_topic": "zigbee2mqtt/Master Bedroom Fan/set/fan_mode", "preset_mode_state_topic": "zigbee2mqtt/Master Bedroom Fan", "preset_mode_value_template": "{{ value_json.fan_mode if value_json.fan_mode in ['on'] else 'None' | default('None') }}", "preset_modes": [ "on" ], "speed_range_max": 3, "speed_range_min": 1, "state_topic": "zigbee2mqtt/Master Bedroom Fan", "state_value_template": "{{ value_json.fan_state }}", "unique_id": "0x048727fffe1c75b8_fan_zigbee2mqtt", "platform": "mqtt" }

[craigwitter]

I also see > 2023-11-10 08:36:32.347 DEBUG (MainThread) [homeassistant.components.mqtt.update] Non-dictionary JSON payload detected after processing payload '33685764' on topic zigbee2mqtt/Master Bedroom Fan

In the MQTT Debug log

[craigwitter]

Upon doing another restart with Z2M in debug mode, the entities went unavaliable at 9:12:35, here is the debug log from that part of theime

info 2023-11-10 09:12:34: MQTT publish: topic 'zigbee2mqtt/Master Bedroom Fan', payload '{"action":null,"autoTimerOff":0,"auxSwitchUniqueScenes":"Disabled","bindingOffToOnSyncLevel":"Disabled","brightnessLevelForDoubleTapDown":2,"brightnessLevelForDoubleTapUp":254,"buttonDelay":"100ms","defaultLed1ColorWhenOff":255,"defaultLed1ColorWhenOn":255,"defaultLed1IntensityWhenOff":101,"defaultLed1IntensityWhenOn":101,"defaultLed2ColorWhenOff":255,"defaultLed2ColorWhenOn":255,"defaultLed2IntensityWhenOff":101,"defaultLed2IntensityWhenOn":101,"defaultLed3ColorWhenOff":255,"defaultLed3ColorWhenOn":255,"defaultLed3IntensityWhenOff":101,"defaultLed3IntensityWhenOn":101,"defaultLed4ColorWhenOff":255,"defaultLed4ColorWhenOn":255,"defaultLed4IntensityWhenOff":101,"defaultLed4IntensityWhenOn":101,"defaultLed5ColorWhenOff":255,"defaultLed5ColorWhenOn":255,"defaultLed5IntensityWhenOff":101,"defaultLed5IntensityWhenOn":101,"defaultLed6ColorWhenOff":255,"defaultLed6ColorWhenOn":255,"defaultLed6IntensityWhenOff":101,"defaultLed6IntensityWhenOn":101,"defaultLed7ColorWhenOff":255,"defaultLed7ColorWhenOn":255,"defaultLed7IntensityWhenOff":101,"defaultLed7IntensityWhenOn":101,"defaultLevelLocal":255,"defaultLevelRemote":255,"deviceBindNumber":0,"dimmingSpeedDownLocal":127,"dimmingSpeedDownRemote":127,"dimmingSpeedUpLocal":127,"dimmingSpeedUpRemote":5,"doubleTapClearNotifications":"Disabled","doubleTapDownToParam56":"Enabled","doubleTapUpToParam55":"Enabled","fanLedLevelType":10,"fan_mode":"high","fan_state":"ON","firmwareUpdateInProgressIndicator":"Enabled","higherOutputInNonNeutral":null,"individual_led_effect":null,"internalTemperature":34,"invertSwitch":"No","last_seen":"2023-11-08T13:50:48-05:00","ledColorWhenOff":255,"ledColorWhenOn":255,"ledIntensityWhenOff":1,"ledIntensityWhenOn":5,"led_effect":null,"linkquality":72,"loadLevelIndicatorTimeout":"Stay On","localProtection":"Disabled","maximumLevel":254,"minimumLevel":1,"nonNeutralAuxLowGear":110,"nonNeutralAuxMediumGear":90,"onOffLedMode":"All","outputMode":"On/Off","overheat":"No Alert","powerType":"Neutral","rampRateOffToOnLocal":127,"rampRateOffToOnRemote":127,"rampRateOnToOffLocal":127,"rampRateOnToOffRemote":127,"remoteProtection":"Disabled","smartBulbMode":"Disabled","state":"ON","stateAfterPowerRestored":255,"switchType":"Aux Switch","update":{"installed_version":33685764,"latest_version":33685764,"state":"idle"},"update_available":false}' debug 2023-11-10 09:12:34: Received Zigbee message from 'Comcast Cable Modem', type 'attributeReport', cluster 'haElectricalMeasurement', data '{"powerFactor":56}' from endpoint 1 with groupID 0 info 2023-11-10 09:12:34: MQTT publish: topic 'zigbee2mqtt/Comcast Cable Modem', payload '{"ac_frequency":60,"current":0.32,"energy":24.62,"last_seen":"2023-11-08T13:49:37-05:00","linkquality":40,"power":21.9,"power_factor":0.56,"power_on_behavior":"on","state":"ON","update":{"installed_version":268513348,"latest_version":268513348,"state":"idle"},"update_available":null,"voltage":119.5}' debug 2023-11-10 09:12:34: Received Zigbee message from 'Washer Energy Monitor', type 'attributeReport', cluster 'haElectricalMeasurement', data '{"powerFactor":31}' from endpoint 1 with groupID 0 info 2023-11-10 09:12:34: MQTT publish: topic 'zigbee2mqtt/Washer Energy Monitor', payload '{"ac_frequency":60,"current":0.04,"energy":8.23,"last_seen":"2023-11-08T13:50:01-05:00","linkquality":43,"power":1.5,"power_factor":0.31,"power_on_behavior":"on","state":"ON","update":{"installed_version":268513348,"latest_version":268513348,"state":"idle"},"update_available":null,"voltage":122.1}' debug 2023-11-10 09:12:36: Received Zigbee message from 'Master Bedroom Fan', type 'attributeReport', cluster 'genOnOff', data '{"onOff":1}' from endpoint 1 with groupID 0 info 2023-11-10 09:12:36: MQTT publish: topic 'zigbee2mqtt/Master Bedroom Fan', payload '{"action":null,"autoTimerOff":0,"auxSwitchUniqueScenes":"Disabled","bindingOffToOnSyncLevel":"Disabled","brightnessLevelForDoubleTapDown":2,"brightnessLevelForDoubleTapUp":254,"buttonDelay":"100ms","defaultLed1ColorWhenOff":255,"defaultLed1ColorWhenOn":255,"defaultLed1IntensityWhenOff":101,"defaultLed1IntensityWhenOn":101,"defaultLed2ColorWhenOff":255,"defaultLed2ColorWhenOn":255,"defaultLed2IntensityWhenOff":101,"defaultLed2IntensityWhenOn":101,"defaultLed3ColorWhenOff":255,"defaultLed3ColorWhenOn":255,"defaultLed3IntensityWhenOff":101,"defaultLed3IntensityWhenOn":101,"defaultLed4ColorWhenOff":255,"defaultLed4ColorWhenOn":255,"defaultLed4IntensityWhenOff":101,"defaultLed4IntensityWhenOn":101,"defaultLed5ColorWhenOff":255,"defaultLed5ColorWhenOn":255,"defaultLed5IntensityWhenOff":101,"defaultLed5IntensityWhenOn":101,"defaultLed6ColorWhenOff":255,"defaultLed6ColorWhenOn":255,"defaultLed6IntensityWhenOff":101,"defaultLed6IntensityWhenOn":101,"defaultLed7ColorWhenOff":255,"defaultLed7ColorWhenOn":255,"defaultLed7IntensityWhenOff":101,"defaultLed7IntensityWhenOn":101,"defaultLevelLocal":255,"defaultLevelRemote":255,"deviceBindNumber":0,"dimmingSpeedDownLocal":127,"dimmingSpeedDownRemote":127,"dimmingSpeedUpLocal":127,"dimmingSpeedUpRemote":5,"doubleTapClearNotifications":"Disabled","doubleTapDownToParam56":"Enabled","doubleTapUpToParam55":"Enabled","fanLedLevelType":10,"fan_mode":"high","fan_state":"ON","firmwareUpdateInProgressIndicator":"Enabled","higherOutputInNonNeutral":null,"individual_led_effect":null,"internalTemperature":34,"invertSwitch":"No","last_seen":"2023-11-08T13:50:48-05:00","ledColorWhenOff":255,"ledColorWhenOn":255,"ledIntensityWhenOff":1,"ledIntensityWhenOn":5,"led_effect":null,"linkquality":72,"loadLevelIndicatorTimeout":"Stay On","localProtection":"Disabled","maximumLevel":254,"minimumLevel":1,"nonNeutralAuxLowGear":110,"nonNeutralAuxMediumGear":90,"onOffLedMode":"All","outputMode":"On/Off","overheat":"No Alert","powerType":"Neutral","rampRateOffToOnLocal":127,"rampRateOffToOnRemote":127,"rampRateOnToOffLocal":127,"rampRateOnToOffRemote":127,"remoteProtection":"Disabled","smartBulbMode":"Disabled","state":"ON","stateAfterPowerRestored":255,"switchType":"Aux Switch","update":{"installed_version":33685764,"latest_version":33685764,"state":"idle"},"update_available":false}' debug 2023-11-10 09:12:36: Received Zigbee message from 'Master Bedroom Fan', type 'attributeReport', cluster 'genOnOff', data '{"onOff":1}' from endpoint 1 with groupID 0 info 2023-11-10 09:12:36: MQTT publish: topic 'zigbee2mqtt/Master Bedroom Fan', payload '{"action":null,"autoTimerOff":0,"auxSwitchUniqueScenes":"Disabled","bindingOffToOnSyncLevel":"Disabled","brightnessLevelForDoubleTapDown":2,"brightnessLevelForDoubleTapUp":254,"buttonDelay":"100ms","defaultLed1ColorWhenOff":255,"defaultLed1ColorWhenOn":255,"defaultLed1IntensityWhenOff":101,"defaultLed1IntensityWhenOn":101,"defaultLed2ColorWhenOff":255,"defaultLed2ColorWhenOn":255,"defaultLed2IntensityWhenOff":101,"defaultLed2IntensityWhenOn":101,"defaultLed3ColorWhenOff":255,"defaultLed3ColorWhenOn":255,"defaultLed3IntensityWhenOff":101,"defaultLed3IntensityWhenOn":101,"defaultLed4ColorWhenOff":255,"defaultLed4ColorWhenOn":255,"defaultLed4IntensityWhenOff":101,"defaultLed4IntensityWhenOn":101,"defaultLed5ColorWhenOff":255,"defaultLed5ColorWhenOn":255,"defaultLed5IntensityWhenOff":101,"defaultLed5IntensityWhenOn":101,"defaultLed6ColorWhenOff":255,"defaultLed6ColorWhenOn":255,"defaultLed6IntensityWhenOff":101,"defaultLed6IntensityWhenOn":101,"defaultLed7ColorWhenOff":255,"defaultLed7ColorWhenOn":255,"defaultLed7IntensityWhenOff":101,"defaultLed7IntensityWhenOn":101,"defaultLevelLocal":255,"defaultLevelRemote":255,"deviceBindNumber":0,"dimmingSpeedDownLocal":127,"dimmingSpeedDownRemote":127,"dimmingSpeedUpLocal":127,"dimmingSpeedUpRemote":5,"doubleTapClearNotifications":"Disabled","doubleTapDownToParam56":"Enabled","doubleTapUpToParam55":"Enabled","fanLedLevelType":10,"fan_mode":"high","fan_state":"ON","firmwareUpdateInProgressIndicator":"Enabled","higherOutputInNonNeutral":null,"individual_led_effect":null,"internalTemperature":34,"invertSwitch":"No","last_seen":"2023-11-08T13:50:48-05:00","ledColorWhenOff":255,"ledColorWhenOn":255,"ledIntensityWhenOff":1,"ledIntensityWhenOn":5,"led_effect":null,"linkquality":72,"loadLevelIndicatorTimeout":"Stay On","localProtection":"Disabled","maximumLevel":254,"minimumLevel":1,"nonNeutralAuxLowGear":110,"nonNeutralAuxMediumGear":90,"onOffLedMode":"All","outputMode":"On/Off","overheat":"No Alert","powerType":"Neutral","rampRateOffToOnLocal":127,"rampRateOffToOnRemote":127,"rampRateOnToOffLocal":127,"rampRateOnToOffRemote":127,"remoteProtection":"Disabled","smartBulbMode":"Disabled","state":"ON","stateAfterPowerRestored":255,"switchType":"Aux Switch","update":{"installed_version":33685764,"latest_version":33685764,"state":"idle"},"update_available":false}' debug 2023-11-10 09:12:37: Received Zigbee message from 'Master Bedroom Fan', type 'attributeReport', cluster 'genOnOff', data '{"onOff":1}' from endpoint 1 with groupID 0 info 2023-11-10 09:12:37: MQTT publish: topic 'zigbee2mqtt/Master Bedroom Fan', payload '{"action":null,"autoTimerOff":0,"auxSwitchUniqueScenes":"Disabled","bindingOffToOnSyncLevel":"Disabled","brightnessLevelForDoubleTapDown":2,"brightnessLevelForDoubleTapUp":254,"buttonDelay":"100ms","defaultLed1ColorWhenOff":255,"defaultLed1ColorWhenOn":255,"defaultLed1IntensityWhenOff":101,"defaultLed1IntensityWhenOn":101,"defaultLed2ColorWhenOff":255,"defaultLed2ColorWhenOn":255,"defaultLed2IntensityWhenOff":101,"defaultLed2IntensityWhenOn":101,"defaultLed3ColorWhenOff":255,"defaultLed3ColorWhenOn":255,"defaultLed3IntensityWhenOff":101,"defaultLed3IntensityWhenOn":101,"defaultLed4ColorWhenOff":255,"defaultLed4ColorWhenOn":255,"defaultLed4IntensityWhenOff":101,"defaultLed4IntensityWhenOn":101,"defaultLed5ColorWhenOff":255,"defaultLed5ColorWhenOn":255,"defaultLed5IntensityWhenOff":101,"defaultLed5IntensityWhenOn":101,"defaultLed6ColorWhenOff":255,"defaultLed6ColorWhenOn":255,"defaultLed6IntensityWhenOff":101,"defaultLed6IntensityWhenOn":101,"defaultLed7ColorWhenOff":255,"defaultLed7ColorWhenOn":255,"defaultLed7IntensityWhenOff":101,"defaultLed7IntensityWhenOn":101,"defaultLevelLocal":255,"defaultLevelRemote":255,"deviceBindNumber":0,"dimmingSpeedDownLocal":127,"dimmingSpeedDownRemote":127,"dimmingSpeedUpLocal":127,"dimmingSpeedUpRemote":5,"doubleTapClearNotifications":"Disabled","doubleTapDownToParam56":"Enabled","doubleTapUpToParam55":"Enabled","fanLedLevelType":10,"fan_mode":"high","fan_state":"ON","firmwareUpdateInProgressIndicator":"Enabled","higherOutputInNonNeutral":null,"individual_led_effect":null,"internalTemperature":34,"invertSwitch":"No","last_seen":"2023-11-08T13:50:48-05:00","ledColorWhenOff":255,"ledColorWhenOn":255,"ledIntensityWhenOff":1,"ledIntensityWhenOn":5,"led_effect":null,"linkquality":72,"loadLevelIndicatorTimeout":"Stay On","localProtection":"Disabled","maximumLevel":254,"minimumLevel":1,"nonNeutralAuxLowGear":110,"nonNeutralAuxMediumGear":90,"onOffLedMode":"All","outputMode":"On/Off","overheat":"No Alert","powerType":"Neutral","rampRateOffToOnLocal":127,"rampRateOffToOnRemote":127,"rampRateOnToOffLocal":127,"rampRateOnToOffRemote":127,"remoteProtection":"Disabled","smartBulbMode":"Disabled","state":"ON","stateAfterPowerRestored":255,"switchType":"Aux Switch","update":{"installed_version":33685764,"latest_version":33685764,"state":"idle"},"update_available":false}'

[Sprinterfreak]

I have a similar problem. A bit further down in the logs it looks like this for me:

2023-11-14 22:35:51.945 ERROR (MainThread) [homeassistant.util.logging] Exception in async_discover when dispatching 'mqtt_discovery_new_update_mqtt': ({'availability': [{'topic': 'zigbee2mqtt/bridge/state'}], 'command_topic': 'zigbee2mqtt/bridge/request/device/ota_update/update', 'device': {'configuration_url': 'http://dockerio.reg.saenet.de:8080/#/device/0x70ac08fffe94eef1/info', 'identifiers': ['zigbee2mqtt_0x70ac08fffe94eef1'], 'manufacturer': 'IKEA', 'model': 'TRADFRI shortcut button (E1812)', 'name': 'EG_Kuche_Arbeitsmodus', 'sw_version': '2.3.015'}, 'device_class': 'firmware', 'entity_category': 'config', 'entity_picture': 'https://github.com/Koenkk/zigbee2mqtt/raw/master/images/logo.png', 'json_attributes_template': '{"in_progress": "{{ value_json[\'update\'][\'state\'] == \'updating\' }}"}', 'json_attributes_topic': 'zigbee2mqtt/EG Küche Arbeitsmodus', 'latest_version_template': "{{ value_json['update']['latest_version'] }}", 'latest_version_topic': 'zigbee2mqtt/EG Küche Arbeitsmodus', 'name': None, 'object_id': 'eg_kuche_arbeitsmodus', 'origin': {'name': 'Zigbee2MQTT', 'sw': '1.33.2', 'url': 'https://www.zigbee2mqtt.io'}, 'payload_install': '{"id": "0x70ac08fffe94eef1"}', 'state_topic': 'zigbee2mqtt/EG Küche Arbeitsmodus', 'unique_id': '0x70ac08fffe94eef1_update_zigbee2mqtt', 'value_template': "{{ value_json['update']['installed_version'] }}", 'platform': 'mqtt'},)
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.11/site-packages/homeassistant/components/mqtt/mixins.py", line 270, in async_discover
    config: DiscoveryInfoType = discovery_schema(discovery_payload)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/homeassistant/lib/python3.11/site-packages/voluptuous/validators.py", line 232, in __call__
    return self._exec((Schema(val) for val in self.validators), v)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/homeassistant/lib/python3.11/site-packages/voluptuous/validators.py", line 355, in _exec
    raise e if self.msg is None else AllInvalid(self.msg, path=path)
  File "/srv/homeassistant/lib/python3.11/site-packages/voluptuous/validators.py", line 351, in _exec
    v = func(v)
        ^^^^^^^
  File "/srv/homeassistant/lib/python3.11/site-packages/voluptuous/schema_builder.py", line 272, in __call__
    return self._compiled([], data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/homeassistant/lib/python3.11/site-packages/voluptuous/schema_builder.py", line 818, in validate_callable
    return schema(data)
           ^^^^^^^^^^^^
  File "/srv/homeassistant/lib/python3.11/site-packages/voluptuous/schema_builder.py", line 272, in __call__
    return self._compiled([], data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/homeassistant/lib/python3.11/site-packages/voluptuous/schema_builder.py", line 595, in validate_dict
    return base_validate(path, iteritems(data), out)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/homeassistant/lib/python3.11/site-packages/voluptuous/schema_builder.py", line 433, in validate_mapping
    raise er.MultipleInvalid(errors)
voluptuous.error.MultipleInvalid: string value is None for dictionary value @ data['name']

So HASS does not seem to allow name to be null.

[mundschenk-at]

@Sprinterfreak This looks like you are running a HA version < 2023.8. If so, you need to upgrade to at least 2023.8.x.

[Sprinterfreak]

@mundschenk-at You where right. Update HA made it work properly and broke other integrations :)

[github-actions[bot]]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 30 days