log error UnicodeDecodeError

[Anto79-ops]

What happened?

Hello,

Seeing this in AD addon on Home Assistant OS (ver: 0.16.6 and 0.16.7 ) logs on HAOS.

thanks

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

2024-09-18 10:43:05.065189 INFO MQTT: Unable to decode MQTT message
2024-09-18 10:43:05.065910 ERROR MQTT: Unable to decode MQTT message, with Traceback: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/appdaemon/plugins/mqtt/mqttplugin.py", line 271, in mqtt_on_message
    payload = payload.decode()
              ^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

2024-09-18 10:47:01.173652 INFO MQTT: Unable to decode MQTT message
2024-09-18 10:47:01.174345 ERROR MQTT: Unable to decode MQTT message, with Traceback: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/appdaemon/plugins/mqtt/mqttplugin.py", line 271, in mqtt_on_message
    payload = payload.decode()
              ^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

2024-09-18 10:50:53.357415 INFO MQTT: Unable to decode MQTT message
2024-09-18 10:50:53.358181 ERROR MQTT: Unable to decode MQTT message, with Traceback: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/appdaemon/plugins/mqtt/mqttplugin.py", line 271, in mqtt_on_message
    payload = payload.decode()
              ^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

2024-09-18 10:54:47.955705 INFO MQTT: Unable to decode MQTT message
2024-09-18 10:54:47.956193 ERROR MQTT: Unable to decode MQTT message, with Traceback: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/appdaemon/plugins/mqtt/mqttplugin.py", line 271, in mqtt_on_message
    payload = payload.decode()
              ^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

2024-09-18 10:55:04.333653 INFO MQTT: Unable to decode MQTT message
2024-09-18 10:55:04.334190 ERROR MQTT: Unable to decode MQTT message, with Traceback: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/appdaemon/plugins/mqtt/mqttplugin.py", line 271, in mqtt_on_message
    payload = payload.decode()
              ^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

2024-09-18 11:00:23.826721 INFO MQTT: Unable to decode MQTT message
2024-09-18 11:00:23.827232 ERROR MQTT: Unable to decode MQTT message, with Traceback: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/appdaemon/plugins/mqtt/mqttplugin.py", line 271, in mqtt_on_message
    payload = payload.decode()
              ^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

2024-09-18 11:01:04.448018 INFO MQTT: Unable to decode MQTT message
2024-09-18 11:01:04.448690 ERROR MQTT: Unable to decode MQTT message, with Traceback: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/appdaemon/plugins/mqtt/mqttplugin.py", line 271, in mqtt_on_message
    payload = payload.decode()
              ^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

2024-09-18 11:05:19.438034 INFO MQTT: Unable to decode MQTT message
2024-09-18 11:05:19.438766 ERROR MQTT: Unable to decode MQTT message, with Traceback: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/appdaemon/plugins/mqtt/mqttplugin.py", line 271, in mqtt_on_message
    payload = payload.decode()
              ^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

2024-09-18 11:09:17.908246 INFO MQTT: Unable to decode MQTT message
2024-09-18 11:09:17.908974 ERROR MQTT: Unable to decode MQTT message, with Traceback: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/appdaemon/plugins/mqtt/mqttplugin.py", line 271, in mqtt_on_message
    payload = payload.decode()
              ^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

More information here https://github.com/hassio-addons/addon-appdaemon/issues/350

Version

0.16.7

Installation type

Home Assistant add-on

Relevant log output

No response

Relevant code in the app or config file that caused the issue

No response

Anything else?

No response

[anandv85]

I'm also seeing the similar issue. This is continuously logging once every 10 seconds. Please let me know if anyone need more information to debug this issue. Thanks!

2024-09-28 20:16:31.285018 INFO AppDaemon: New client Admin Client connected
2024-09-28 20:16:52.383142 INFO MQTT: Unable to decode MQTT message
2024-09-28 20:16:52.384120 ERROR MQTT: Unable to decode MQTT message, with Traceback: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/appdaemon/plugins/mqtt/mqttplugin.py", line 271, in mqtt_on_message
    payload = payload.decode()
              ^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

2024-09-28 20:16:52.469429 INFO MQTT: Unable to decode MQTT message
2024-09-28 20:16:52.469850 ERROR MQTT: Unable to decode MQTT message, with Traceback: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/appdaemon/plugins/mqtt/mqttplugin.py", line 271, in mqtt_on_message
    payload = payload.decode()
              ^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

2024-09-28 20:17:11.707325 INFO AppDaemon: Client disconnection from Admin Client
2024-09-28 20:17:22.250166 INFO MQTT: Unable to decode MQTT message
2024-09-28 20:17:22.251115 ERROR MQTT: Unable to decode MQTT message, with Traceback: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/appdaemon/plugins/mqtt/mqttplugin.py", line 271, in mqtt_on_message
    payload = payload.decode()
              ^^^^^^^^^^^^^^^^

[Anto79-ops]

Hi @hacscred

sorry, im not sure I quite understand (probably because im just a user of AD). I'm currently using AD as an Addon in Home Assistant, and its a supported addon.

Not sure how I can use it as a standalone, you mean in docker for example?

thanks

[Anto79-ops]

still happening in my logs.

[OptimusGREEN]

same issue here too.

[Anto79-ops]

@OptimusGREEN hopefully a new release is coming soon, from what I hear is in the works.

[acockburn]

On the face of it this doesn't look like anything specific to the HASS environment, but this is basically an issue decoding payload which could be literally anything. Some details as to what you are actually doing with MQTT to cause this issue would be helpful.

[OptimusGREEN]

The log doesn't state which topics are related to the issue so it is somewhat impossible to tell. It would be good if when the issue arises that the mqtt module mentioned the naughty topic in the log.

[Anto79-ops]

thanks @acockburn

using this, in my case: https://github.com/xaf/qolsysgw

[acockburn]

OK, we are buried in the current release and we won't get to a resolution of this issue before we get that out but lets see if we can add some additional debugging in this area which shouldn't be too big a deal so that we can start to track it down in a future release (plan is to do more smaller releases more often moving forwards).

[jsl12]

This type of error is caused by an MQTT message having a binary payload, which causes decoding it into utf-8 to fail.

Unfortunately, really fixing this is going to be kind of involved. We have many plans for future work, and redoing some of the MQTT mechanics is in there somewhere.

For the short term though, I'll commit to at least improving the error message so that it shows the offending topic.

[jsl12]

After looking at the code, there's already a feature to cover this, and there's already a better error message in the dev branch.

Unfortunately, you do have to already know what topic has the binary data on it, but you can do this:

    self.add_mqtt_binary("my_topic")

Which will prevent AppDaemon from trying to decode that payload as text

[Anto79-ops]

hey all, happy to be using the new AD! Indeed the error messaging has improved for the new version. I'm now able to see what problematic payload is causing the original log:

2025-06-04 08:41:26.362384 INFO MQTT: Unable to decode MQTT message from topic frigate/shed/person/snapshot, ignoring message
2025-06-04 08:41:26.362736 ERROR MQTT: Unable to decode MQTT message from topic frigate/shed/person/snapshot, with error: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

Here's what im a little confused about, the log is for another application (Frigate) and a photo snapshot, which has nothing to do with AD. So, it seems AD is trying to decode MQTT topics that are not associated with it or any topics in the MQTT broker.

Just wondering if this is normal? thanks @jsl12

[jsl12]

Hi there, glad to hear you like some of the changes. This is normal and depends on what MQTT topics are being subscribed to by AppDaemon, which is configured in appdaemon.yaml.

In general, MQTT is a pub/sub service, so subscribers get sent any messages that are published on topics they're subscribed to. Frigate is publishing messages with pictures on a certain topic, and AppDaemon must be subscribed to it.

It's a common pattern for users to subscribe to ALL MQTT messages and then only act on the ones they're interested in, so I'd imagine that's the kind of situation you're in.

[Anto79-ops]

got it, Ok

I think what I need to do is add the relevent lines into AD to listen only to the one intergration that im using it for. In this case, Qolsys Gateway.

for example,

qolsys/qolsys_panel/event

or even everything after

qolsys/

is there some sort documentation you can refer to me to get this setup?

thanks!!

[jsl12]

In that case, you'd subscribe to qolsys/# to get everything under that topic. For example, I subscribe to zigbee2mqtt/# in my setup

Check this site for more information about MQTT wildcards

[Anto79-ops]

oh thats a great idea, I should add it to my Zigbee2mqtt also, as I think that also gives some logs sometimes about other devices. With regards to AD and Qolsys gateway, something like this?

so I added this:

      topics:
        - qolsys/#

to this:

apps:
  - app_class: MQTT
    app_name: mqtt
    config:
      broker: <your_mqtt_broker_url>
      port: <your_mqtt_broker_port>
      topics:
        - qolsys/#

EDIT: The above does not seem to work, as the extra topics are still trying to be decoded.

EDIT2: oh, nevermind, it seems to be this:

      client_topics:
        - qolsys/#

thank you!