avahi
avahi copied to clipboard
avahi-browse prints invalid UTF-8 for BluetoothAddress property sent by Apple TV
A bug was reported against our project (MAAS), which parses the output of avahi-browse --all --resolve --no-db-lookup --parsable --no-fail
. In the bug, the following error occurred:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc8 in position 9122: invalid continuation byte
After triaging the output, I determined that the following portion of the output was the problem:
>>> hex(9122)
'0x23a2'
$ xxd avahi.log
...
00002280: 443a 3432 3a45 333a 3039 220a 3d3b 656e D:42:E3:09".=;en
00002290: 7333 333b 4950 7636 3b41 7070 6c65 5c30 s33;IPv6;Apple\0
000022a0: 3332 5456 3b5f 6d65 6469 6172 656d 6f74 32TV;_mediaremot
000022b0: 6574 762e 5f74 6370 3b6c 6f63 616c 3b41 etv._tcp;local;A
000022c0: 7070 6c65 2d54 562e 6c6f 6361 6c3b 3139 pple-TV.local;19
000022d0: 322e 3136 382e 312e 3831 3b34 3931 3630 2.168.1.81;49160
000022e0: 3b22 4c6f 6361 6c41 6972 506c 6179 5265 ;"LocalAirPlayRe
000022f0: 6365 6976 6572 5061 6972 696e 6749 6465 ceiverPairingIde
00002300: 6e74 6974 793d 3544 4239 3132 3037 2d44 ntity=5DB91207-D
00002310: 3333 362d 3445 4542 2d41 3534 332d 3538 336-4EEB-A543-58
00002320: 3131 4541 4237 3636 4643 2220 2253 7973 11EAB766FC" "Sys
00002330: 7465 6d42 7569 6c64 5665 7273 696f 6e3d temBuildVersion=
00002340: 3135 4b36 3030 2220 2255 6e69 7175 6549 15K600" "UniqueI
00002350: 6465 6e74 6966 6965 723d 3744 4633 4241 dentifier=7DF3BA
00002360: 4637 2d33 3238 462d 3431 3645 2d42 4330 F7-328F-416E-BC0
00002370: 322d 4135 3636 3345 3637 3342 4238 2220 2-A5663E673BB8"
00002380: 224e 616d 653d 4170 706c 6520 5456 2220 "Name=Apple TV"
00002390: 2242 6c75 6574 6f6f 7468 4164 6472 6573 "BluetoothAddres
000023a0: 733d c869 cd42 e309 2220 2241 6c6c 6f77 s=.i.B.." "Allow
000023b0: 5061 6972 696e 673d 5945 5322 2022 4d6f Pairing=YES" "Mo
000023c0: 6465 6c4e 616d 653d 220a 3d3b 656e 7333 delName=".=;ens3
...
$ cat avahi.log
...
=;ens33;IPv6;Apple\032TV;_mediaremotetv._tcp;local;Apple-TV.local;192.168.1.81;49160;"LocalAirPlayReceiverPairingIdentity=5DB91207-D336-4EEB-A543-5811EAB766FC" "SystemBuildVersion=15K600" "UniqueIdentifier=7DF3BAF7-328F-416E-BC02-A5663E673BB8" "Name=Apple TV" "BluetoothAddress=<JUNK>" "AllowPairing=YES" "ModelName="
=;ens33;IPv4;Apple\032TV;_mediaremotetv._tcp;local;Apple-TV.local;192.168.1.81;49160;"LocalAirPlayReceiverPairingIdentity=5DB91207-D336-4EEB-A543-5811EAB766FC" "SystemBuildVersion=15K600" "UniqueIdentifier=7DF3BAF7-328F-416E-BC02-A5663E673BB8" "Name=Apple TV" "BluetoothAddress=<JUNK>" "AllowPairing=YES" "ModelName="
...
That is, an Apple TV device sent a BluetoothAddress=<binary>
property, which avahi-browse
did not escape before printing.
It seems to me that avahi-browse
should document the format of its --parsable
output, including expected escape sequences and character encoding, in order to prevent issues like this.
In mDNS, DNS names are considered to be UTF-8.
TXT record values however have no requirement to be UTF-8, and in many cases are in fact binary such as we see here.
I think the only way to make this work is to parse the TXT record section of the output as a byte stream, and only convert to UTF8 later if you are looking to output a specific property that is known as UTF8 (I think in the case of MAAS however, probably you are ignoring the TXT records).
Remember, though, the other data such as the DNS/service names should still be parsed as UTF-8. So you'd have to split parse the string.
As a side note, avahi-browse also runs the output of the TXT section through avahi_escape_label which escapes " characters (0x5C) with a . This is also probably not UTF-8 safe and maybe should be reconsidered - i.e. perhaps we should in parseable format print the TXT records as hex or with a length specifier at the start.
I think if there are any non-ASCII characters in the output, they should be escaped, similar to what you would see in a JSON string. For example, I see this when I run the command on my laptop:
+;wlp4s0;IPv4;HP\032Color\032LaserJet\032CP2025dn\032\0408DD8CD\041;_http._tcp;local
I'm not sure who is responsible for escaping that string, but it looks like \<three-zero-padded-decimal-digits>
could be used to represent any arbitrary non-printable characters in the output. (though escaping spaces seems a bit overzealous...)
I don't think binary values should ever be printed directly; this goes against the spirit of --parsable
since arbitrary binary strings could contain newlines, and could even be used to attack the parser.
Looks like we need a modified version of avahi_string_list_to_string which does a similar escape to avahi_escape_label that is used on the hostnames.
My name on
Should be fixed by this PR: https://github.com/lathiat/avahi/pull/206
If someone can confirm that would be great
I think this issue can be closed.
Strangely it seems I just ran into this issue on the Raspberry Pi 5 with the latest Raspberry Pi OS Lite, trying to get output from Python3 subprocess.
version: avahi-browse 0.8
avahi-browse --all --resolve --no-db-lookup --parsable --no-fail -t
Error running command: 'utf-8' codec can't decode bytes in position 4229-4230: invalid continuation byte
In the end I had to bring in chardet
to handle it.
command = ["avahi-browse","--parsable","--all","--ignore-local","--resolve","--no-fail","--no-db-lookup","--terminate"]
avahi_result = subprocess.check_output(command)
encoding = chardet.detect(avahi_result)
print("chardet: encoding: " + str(encoding))
output = avahi_result.decode(str(encoding['encoding']),"ignore")
Could you paste the output of avahi-browse
with its hexdump? That part of the code is fuzzed with all sort of strings and the result is checked to make sure it's utf-8 so I'm not sure where it comes from.
There are some indications the issue comes from an Apple Homepod Mini.
- I know the code worked before. One of the changes on my network is the introduction on the homepod.
- In the text output, there are some strange characters on the Homepod output:
=;wlan0;IPv6;Apple\032BorderRouter\032\035F6A2;_meshcop._udp;local;Living-Room.local;192.168.0.236;49154;"dn=DefaultDomain" "bb=ð¿" "sq=F" "pt=\022a·R" "at=\000\000\000\000\000\000\000\000" "sb=\000\000\001±" "dd=Z“^Róæö¢" "xa=Z“^Róæö¢" "tv=1.3.0" "xp=\007~6ô°èM³" "nn=MyHome307535120" "mn=BorderRouter" "vn=Apple Inc." "rv=1"
=;wlan0;IPv4;Apple\032BorderRouter\032\035F6A2;_meshcop._udp;local;Living-Room.local;192.168.0.236;49154;"dn=DefaultDomain" "bb=ð¿" "sq=F" "pt=\022a·R" "at=\000\000\000\000\000\000\000\000" "sb=\000\000\001±" "dd=Z“^Róæö¢" "xa=Z“^Róæö¢" "tv=1.3.0" "xp=\007~6ô°èM³" "nn=MyHome307535120" "mn=BorderRouter" "vn=Apple Inc." "rv=1"
Is there a way that I can share the binary output of that device only? I have a hex dump of the entire output, but for privacy and security reasons I'm hesitant to share that online.
Allright, I cut the hex down a bit, I think I can share this:
2b3b776c616e303b495076363b486f6d65506f6453656e736f725c3033323739343436393b5f6861702e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b486f6d65506f6453656e736f725c3033323739343436393b5f6861702e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b4170706c655c303332426f72646572526f757465725c3033325c303335463641323b5f6d657368636f702e5f7564703b6c6f63616c0a2b3b776c616e303b495076343b4170706c655c303332426f72646572526f757465725c3033325c303335463641323b5f6d657368636f702e5f7564703b6c6f63616c0a2b3b776c616e303b495076363b356139333565353266336536663661323b5f7472656c2e5f7564703b6c6f63616c0a2b3b776c616e303b495076343b356139333565353266336536663661323b5f7472656c2e5f7564703b6c6f63616c0a2b3b776c616e303b495076363b4c6976696e675c303332526f6f6d3b5f7372706c2d746c732e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b4c6976696e675c303332526f6f6d3b5f7372706c2d746c732e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b37302d33352d36302d36335c2e315c3033324c6976696e675c303332526f6f6d3b5f736c6565702d70726f78792e5f7564703b6c6f63616c0a2b3b776c616e303b495076343b37302d33352d36302d36335c2e315c3033324c6976696e675c303332526f6f6d3b5f736c6565702d70726f78792e5f7564703b6c6f63616c0a2b3b776c616e303b495076363b4e4153343b5f776562646176732e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b4e4153343b5f776562646176732e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b4e4153343b5f7765626461762e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b4e4153343b5f7765626461762e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b4e4153343b5f616469736b2e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b4e4153343b5f616469736b2e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b4e4153343b5f736674702e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b4e4153343b5f736674702e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b4e4153343b5f6166706f7665727463702e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b4e4153343b5f6166706f7665727463702e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b4e6f6b69615c3033324e39353b5f636f6d70616e696f6e2d6c696e6b2e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b4c6976696e675c303332526f6f6d3b5f636f6d70616e696f6e2d6c696e6b2e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b4e6f6b69615c3033324e39353b5f636f6d70616e696f6e2d6c696e6b2e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b4c6976696e675c303332526f6f6d3b5f636f6d70616e696f6e2d6c696e6b2e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b4638344438393543464131345c3036344e6f6b69615c3033324e39353b5f72616f702e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b4643383042443443443137335c3036344d656c6b5c303332617564696f3b5f72616f702e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b4134434639394236333538335c3036344c6976696e675c303332526f6f6d3b5f72616f702e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b4643383042443443443137335c3036344d656c6b5c303332617564696f3b5f72616f702e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b4638344438393543464131345c3036344e6f6b69615c3033324e39353b5f72616f702e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b4134434639394236333538335c3036344c6976696e675c303332526f6f6d3b5f72616f702e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b4e6f6b69615c3033324e39353b5f616972706c61792e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b4d656c6b5c303332617564696f3b5f616972706c61792e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b4c6976696e675c303332526f6f6d3b5f616972706c61792e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b4d656c6b5c303332617564696f3b5f616972706c61792e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b4e6f6b69615c3033324e39353b5f616972706c61792e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b4c6976696e675c303332526f6f6d3b5f616972706c61792e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b4e4153343b5f736d622e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b4445563b5f736d622e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b4e4153343b5f736d622e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b4445563b5f736d622e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b626564726f6f6d5c2e6c6f63616c5c3033325c30343043616e646c655c3034313b5f687474702e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b6d656c6b5c3033325c30343043616e646c655c3034313b5f687474702e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b4e4153343b5f687474702e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b6465765c2e6c6f63616c5c3033325c30343043616e646c655c3034313b5f687474702e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b626564726f6f6d5c2e6c6f63616c5c3033325c30343043616e646c655c3034313b5f687474702e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b6d656c6b5c3033325c30343043616e646c655c3034313b5f687474702e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b4e4153343b5f687474702e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b6465765c2e6c6f63616c5c3033325c30343043616e646c655c3034313b5f687474702e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b43616e646c654d5154542d626564726f6f6d3b5f6d7174742e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b43616e646c654d5154542d6d656c6b3b5f6d7174742e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b43616e646c654d5154542d6465763b5f6d7174742e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b43616e646c654d5154542d626564726f6f6d3b5f6d7174742e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b43616e646c654d5154542d6d656c6b3b5f6d7174742e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b43616e646c654d5154542d6465763b5f6d7174742e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b6d656c6b5c3033325c30343043616e646c655c3034313b5f6465766963652d696e666f2e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b4e4153343b5f6465766963652d696e666f2e5f7463703b6c6f63616c0a2b3b776c616e303b495076363b4445563b5f6465766963652d696e666f2e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b6d656c6b5c3033325c30343043616e646c655c3034313b5f6465766963652d696e666f2e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b4e4153343b5f6465766963652d696e666f2e5f7463703b6c6f63616c0a2b3b776c616e303b495076343b4445563b5f6465766963652d696e666f2e5f7463703b6c6f63616c0a3d3b776c616e303b495076363b486f6d65506f6453656e736f725c3033323739343436393b5f6861702e5f7463703b6c6f63616c3b4c6976696e672d526f6f6d2e6c6f63616c3b3139322e3136382e302e3233363b35363232373b2273683d586135544a673d3d22202263693d313022202273663d3022202273233d3422202270763d312e312220226d643d486f6d65506f6422202269643d35413a36423a45313a39393a41333a443222202266663d3222202263233d31220a3d3b776c616e303b495076343b486f6d65506f6453656e736f725c3033323739343436393b5f6861702e5f7463703b6c6f63616c3b4c6976696e672d526f6f6d2e6c6f63616c3b3139322e3136382e302e3233363b35363232373b2273683d586135544a673d3d22202263693d313022202273663d3022202273233d3422202270763d312e312220226d643d486f6d65506f6422202269643d35413a36423a45313a39393a41333a443222202266663d3222202263233d31220a3d3b776c616e303b495076363b4170706c655c303332426f72646572526f757465725c3033325c303335463641323b5f6d657368636f702e5f7564703b6c6f63616c3b4c6976696e672d526f6f6d2e6c6f63616c3b3139322e3136382e302e3233363b34393135343b22646e3d44656661756c74446f6d61696e22202262623df0bf22202273713d4622202270743d5c30323261b75222202261743d5c3030305c3030305c3030305c3030305c3030305c3030305c3030305c30303022202273623d5c3030305c3030305c303031b122202264643d5a935e52f3e6f6a222202278613d5a935e52f3e6f6a222202274763d312e332e3022202278703d5c3030377e36f4b0e84db32220226e6e3d4d79486f6d653330373533353132302220226d6e3d426f72646572526f75746572222022766e3d4170706c6520496e632e22202272763d31220a3d3b776c616e303b495076343b4170706c655c303332426f72646572526f757465725c3033325c303335463641323b5f6d657368636f702e5f7564703b6c6f63616c3b4c6976696e672d526f6f6d2e6c6f63616c3b3139322e3136382e302e3233363b34393135343b22646e3d44656661756c74446f6d61696e22202262623df0bf22202273713d4622202270743d5c30323261b75222202261743d5c3030305c3030305c3030305c3030305c3030305c3030305c3030305c30303022202273623d5c3030305c3030305c303031b122202264643d5a935e52f3e6f6a222202278613d5a935e52f3e6f6a222202274763d312e332e3022202278703d5c3030377e36f4b0e84db32220226e6e3d4d79486f6d653330373533353132302220226d6e3d426f72646572526f75746572222022766e3d4170706c6520496e632e22202272763d31220a3d3b776c616e303b495076363b356139333565353266336536663661323b5f7472656c2e5f7564703b6c6f63616c3b4c6976696e672d526f6f6d2e6c6f63616c3b3139322e3136382e302e3233363b35373931303b2278703d5c3030377e36f4b0e84db322202278613d5a935e52f3e6f6a2220a3d3b776c616e303b495076343b356139333565353266336536663661323b5f7472656c2e5f7564703b6c6f63616c3b4c6976696e672d526f6f6d2e6c6f63616c3b3139322e3136382e302e3233363b35373931303b2278703d5c3030377e36f4b0e84db322202278613d5a935e52f3e6f6a2220a3d3b776c616e303b495076363b4c6976696e675c303332526f6f6d3b5f7372706c2d746c732e5f7463703b6c6f63616c3b4c6976696e672d526f6f6d2e6c6f63616c3b3139322e3136382e302e3233363b3835333b227870616e69643d3737653336663462306538346462332220226469643d656362373866333063653338656331382220227069643d37363632666334636337326433316161222022646e3d6f70656e7468726561642e7468726561642e686f6d652e617270612e220a2b3b776c616e303b495076343b43616e646c655c30333263616d6572615c3033326b6161733b5f7765627468696e672e5f7463703b6c6f63616c0a2b3b6c6f3b495076343b43616e646c655c30333263616d6572615c3033326b6161733b5f7765627468696e672e5f7463703b6c6f63616c0a3d3b776c616e303b495076343b4c6976696e675c303332526f6f6d3b5f7372706c2d746c732e5f7463703b6c6f63616c3b4c6976696e672d526f6f6d2e6c6f63616c3b3139322e3136382e302e3233363b3835333b227870616e69643d3737653336663462306538346462332220226469643d656362373866333063653338656331382220227069643d37363632666334636337326433316161222022646e3d6f70656e7468726561642e7468726561642e686f6d652e617270612e220a3d3b776c616e303b495076363b37302d33352d36302d36335c2e315c3033324c6976696e675c303332526f6f6d3b5f736c6565702d70726f78792e5f7564703b6c6f63616c3b4c6976696e672d526f6f6d2e6c6f63616c3b3139322e3136382e302e3233363b35303239363b0a3d3b776c616e303b495076343b37302d33352d36302d36335c2e315c3033324c6976696e675c303332526f6f6d3b5f736c6565702d70726f78792e5f7564703b6c6f63616c3b4c6976696e672d526f6f6d2e6c6f63616c3b3139322e3136382e302e3233363b35303239363b0a3d3b776c616e303b495076363b4e4153343b5f776562646176732e5f7463703b6c6f63616c3b4e4153342e6c6f63616c3b666430373a376533363a663462303a
This was made by:
avahi_result = subprocess.check_output(command)
print(".hex(): " + str(avahi_result.hex() ))
The culprit seems to be
>>> b[3420:3434]
b'"bb=\xf0\xbf" "sq=F"'
With the master branch of avahi it turns into "bb=\240\191"
as far as I can see.
version: avahi-browse 0.8
FWIW With v0.8 this isn't reproducible either. I had to downgrade avahi to v0.7
to see those two bytes in the output at the end:
v0.7
00000000 2b 3b 65 74 68 31 3b 49 50 76 34 3b 58 3b 5f 71 |+;eth1;IPv4;X;_q|
00000010 6f 74 64 2e 5f 74 63 70 3b 6c 6f 63 61 6c 0a 3d |otd._tcp;local.=|
00000020 3b 65 74 68 31 3b 49 50 76 34 3b 58 3b 5f 71 6f |;eth1;IPv4;X;_qo|
00000030 74 64 2e 5f 74 63 70 3b 6c 6f 63 61 6c 3b 68 6f |td._tcp;local;ho|
00000040 73 74 2d 35 2e 6c 6f 63 61 6c 3b 31 39 32 2e 31 |st-5.local;192.1|
00000050 36 38 2e 35 36 2e 33 3b 31 32 33 3b 22 62 3d f0 |68.56.3;123;"b=.|
00000060 bf 22 0a |.".|
v0.8
+;eth1;IPv4;X;_qotd._tcp;local
=;eth1;IPv4;X;_qotd._tcp;local;host-5.local;192.168.56.3;123;"b=\240\191"
I'm not sure what the latest Raspberry Pi OS Lite ships but it's certainly not v0.8.
That's odd. The version comes from running avahi-browse -V
. I just double checked :-)
It was installed with avahi-utils --no-install-recommends
.
Running that again gives:
avahi-utils is already the newest version (0.8-10).
uname -a:
Linux melk 6.1.0-rpi8-rpi-2712 #1 SMP PREEMPT Debian 1:6.1.73-1+rpt1 (2024-01-25) aarch64 GNU/Linux
aarch64
Got it. That explains everything. char is unsigned there. Generally currently avahi doesn't escape strings properly when chars are unsigned so in its current form it works on x86_64 and i386 where if (*p < 32)
is true for \xf0
and \xbf
. Looking at https://github.com/avahi/avahi/commit/4a5454fdf7e012b88904c2bd5423e5ccdf2ede92 it seems it was an unintentional side effect.