bleak BluetoothLEAdvertisementDataSection::data is corrupt, cannot be read with WinRT DataReader

bleak version: 0.13.0
Python version: 3.9.9
Operating System: Microsoft Windows 10 Home (10.0.19043 N/A Build 19043)
BlueZ version (bluetoothctl -v) in case of Linux: N/A

Description

I am trying to access the BLE ServiceData (Data type: 0x16) from a BLE advertisement. I noticed that this works okay with Bleak v0.12.0 but running into trouble with Bleak v0.13.0. Everything else is the same, tried to uninstall and downgrade to v0.12.0 and the below code works but upgrading to v0.13.0 breaks DataReader.from_buffer() causing it to throw Runtime Error (not much information other than an error message stating "the parameter is incorrect").

What I Did

# Requirements
# pip3 install winrt
# pip3 install bleak==0.13 // fails with this version
# pip3 install bleak==0.12 // works with this version

import asyncio
from bleak import BleakScanner
from winrt.windows.storage.streams import DataReader


# See https://docs.microsoft.com/en-us/uwp/api/windows.devices.bluetooth.advertisement.bluetoothleadvertisement?view=winrt-19041
def detection_callback(device, advertisement_data):
        for data_section in device.details.advertisement.data_sections:
            print("Data type: ", data_section.data_type)
            print("Data length: ", data_section.data.length)
            dataReader = DataReader.from_buffer(data_section.data)
            print("Data: ", " ".join([hex(dataReader.read_byte()) for i in range(data_section.data.length)]))

async def run():
    scanner = BleakScanner(filters={"scanning_mode":"passive"})
    scanner.register_detection_callback(detection_callback)
    await scanner.start()
    await asyncio.sleep(5)
    await scanner.stop()

loop = asyncio.get_event_loop()
loop.run_until_complete(run())

Dec 30 '21 02:12 aravindsvu

Bleak already decodes the service data for you. There is no need to call OS-specific functions in your code. You can access it in detection_callback() with advertisement_data.service_data.

https://github.com/hbldh/bleak/blob/b644ae28e44a072879c57bcc6a002affeb34f600/bleak/backends/winrt/scanner.py#L128-L132

Dec 30 '21 02:12 dlech

@dlech thanks for the quick response. I did try it but for some reason when I use detection_callback() I get empty dict when I print advertisement_data.service_data. Also weirdly the data sections only has the data types 9 and 10 (no 22) which might explain why the service_data is empty. But if I use the synchronous call to discover the devices, then I do see the data sections with data types 1 and 22 (but not 9 and 10).

I wonder in the case of asynchronous scan, the data included in the SCAN_RESP is somehow overwriting/hiding the data in the advertisement packets. I don't know what data is included in the SCAN_RESP for my BLE peripheral, but given 9 and 10 are "complete local name" and "tx power level" see Bluetooth GAP, I suspect this might be the case. Usually this information is included in the scan response if there is not sufficient space in the advertisement packet itself. I did try to set the scanning type to be passive but that had no impact.

Below is the code and result with bleak 0.13.0 (but the same behavior with 0.12.0 as well) when using the synchronous vs asynchronous calls to scan over BLE. The same device was advertising in both cases (only showing my BLE peripheral, clipped out other device data).

Blocking Scan:

# requirements
# pip3 install winrt
# pip3 install bleak
import asyncio
from bleak import BleakScanner

async def run():
    devices = await BleakScanner.discover()
    for d in devices:
        print("Address: %s RSSI: %d" % (d.address, d.rssi))
        for data in d.details.advertisement.data_sections:
            print("Data type: ", data.data_type)
            print("Data length: ", data.data.length)

loop = asyncio.get_event_loop()
loop.run_until_complete(run())

Output seen:

Address: C3:55:CD:85:98:29 RSSI: -83
Data type:  1
Data length:  1
Data type:  22
Data length:  22

Non blocking scan:

# Requirements
# pip3 install winrt
# pip3 install bleak

import asyncio
from bleak import BleakScanner


# See https://docs.microsoft.com/en-us/uwp/api/windows.devices.bluetooth.advertisement.bluetoothleadvertisement?view=winrt-19041
def detection_callback(device, advertisement_data):
        print("Address: %s RSSI: %d" % (device.address, device.rssi))
        print("Service data: ", advertisement_data.service_data)
        for data_section in device.details.advertisement.data_sections:
            print("Data type: ", data_section.data_type)
            print("Data length: ", data_section.data.length)

async def run():
    scanner = BleakScanner(filters={"scanning_mode":"passive"})
    scanner.register_detection_callback(detection_callback)
    await scanner.start()
    await asyncio.sleep(5)
    await scanner.stop()

loop = asyncio.get_event_loop()
loop.run_until_complete(run())

Output seen:

Address: C3:55:CD:85:98:29 RSSI: -68
Service data:  {}
Data type:  10
Data length:  1
Data type:  9
Data length:  4
Address: C3:55:CD:85:98:29 RSSI: -69
Service data:  {}
Data type:  10
Data length:  1
Data type:  9
Data length:  4
Address: C3:55:CD:85:98:29 RSSI: -84
Service data:  {}
Data type:  10
Data length:  1
Data type:  9
Data length:  4
Address: C3:55:CD:85:98:29 RSSI: -72
Service data:  {}
Data type:  10
Data length:  1
Data type:  9
Data length:  4
Address: C3:55:CD:85:98:29 RSSI: -74
Service data:  {}
Data type:  10
Data length:  1
Data type:  9
Data length:  4
Address: C3:55:CD:85:98:29 RSSI: -69
Service data:  {}
Data type:  10
Data length:  1
Data type:  9
Data length:  4
Address: C3:55:CD:85:98:29 RSSI: -67
Service data:  {}
Data type:  10
Data length:  1
Data type:  9
Data length:  4
Address: C3:55:CD:85:98:29 RSSI: -68
Service data:  {}
Data type:  10
Data length:  1
Data type:  9
Data length:  4
Address: C3:55:CD:85:98:29 RSSI: -75
Service data:  {}
Data type:  10
Data length:  1
Data type:  9
Data length:  4
Address: C3:55:CD:85:98:29 RSSI: -74
Service data:  {}
Data type:  10
Data length:  1
Data type:  9
Data length:  4

Dec 30 '21 03:12 aravindsvu

Windows does indeed send the scan response data in a separate event (unlike other OSes). I made a fix for this in https://github.com/hbldh/bleak/commit/3f26be60c3e69d61c5327a0fe0bbf0aee20f1456 but it hasn't been released yet. You can pip install https://github.com/hbldh/bleak/archive/refs/heads/develop.zip to try it.

Dec 30 '21 04:12 dlech

Ok thanks for pointing me to the fix for the issue with the asynchronous calls. I will give it a go, once I get a chance.

However just to make sure the original issue I reported doesn't get side tracked, any thoughts on why winrt.windows.storage.streams.DataReader.from_buffer() is throwing run time error when passing the winrt.windows.storage.streams.IBuffer object returned by BluetoothLEAdvertisementDataSection::data in the bleak version 0.13.0 but not in the version 0.12.0?

Since I need access to the service data i have downgraded to 0.12.0 for now which works. Once I test your fix for getting the service data from the asynchronous calls, I would be able to upgrade But the original issue reported here affects the ability to read any data section from the advertisement payload, so might be worth checking what's causing this failure in 0.13.0.

Dec 30 '21 06:12 aravindsvu

In Bleak 0.13, we switched to the bleak-winrt package which contains many fixes and improvements to the unmaintained winrt package. One of these is that IBuffer now implements the Python buffer protocol, so you can use it directly with things like struct.unpack() without having to use a DataReader.

Although, I would still expect that DataReader would keep working too. So if you are really interested in fixing it, we should add a test here to reproduce the issue.

Dec 30 '21 16:12 dlech

scanner = BleakScanner(filters={"scanning_mode":"passive"})

FYI, passive mode prevents the device from sending SCAN_RSP data which is why all of the advertisement packets look the same in your output.

Dec 30 '21 23:12 dlech

FYI, passive mode prevents the device from sending SCAN_RSP data which is why all of the advertisement packets look the same in your output.

This is what I was hoping for but the asynchronous scan doesn't seem to be suppressing the SCAN_RSP as I have shown in the example code above. The results contain the data types 9 and 10 which I believe are from the SCAN_RSP for my test BLE peripheral.

So if you are really interested in fixing it, we should add a test here to reproduce the issue.

I noticed you have added a test, thanks.

One of these is that IBuffer now implements the Python buffer protocol, so you can use it directly with things like struct.unpack() without having to use a DataReader.

This is great, thanks for the tip. This works well for me on 013.0. This should help make the code more independent of the underlying platform.

# requirements
# pip3 install bleak
import asyncio
from bleak import BleakScanner
import struct

async def run():
    devices = await BleakScanner.discover()
    for d in devices:
          print("Address: %s RSSI: %d" % (d.address, d.rssi))
          for data in d.details.advertisement.data_sections:
              advData = [dt[0] for dt in struct.iter_unpack('<B', data.data)]
              print("Data type: ", data.data_type)
              print("Length: ", data.data.length)
              print ("Data: ", " ".join([hex(dt) for dt in advData]))
                    
loop = asyncio.get_event_loop()
loop.run_until_complete(run())

result:

Address: CD:40:57:99:5C:F6 RSSI: -70
Data type:  1
Length:  1
Data:  0x6
Data type:  22
Length:  22
Data:  0x0 0xfe 0x0 0xcd 0x3c 0xaf 0xf2 0x2d 0x4e 0x3f 0x27 0xa5 0x73 0xc2 0x9c 0x76 0x62 0x39 0x66 0x30 0x30 0x6

Dec 31 '21 02:12 aravindsvu

advData = [dt[0] for dt in struct.iter_unpack('<B', data.data)]

If you just want to convert to bytes...

advData = bytes(data.data)

This is what I was hoping for but the asynchronous scan doesn't seem to be suppressing the SCAN_RSP as I have shown in the example code above. The results contain the data types 9 and 10 which I believe are from the SCAN_RSP for my test BLE peripheral.

Bleak v0.13.0 is broken in that it doesn't differentiate between the SCAN_RSP and other advertising data. Since SCAN_RSP is received last, it overwrites the other data. This is why details only contains the SCAN_RSP.

The detection_callback() in v0.13.0, on the other hand, should be seeing both advertisement types, but only if active scanning.

Both of these have changed in the develop branch. details will now be a named tuple that contains the last received of both advertisement types and detection_callback() should merge the info from both advertisement types.

Dec 31 '21 03:12 dlech

I'm not super familiar with the inner workings of Bleak, but I am getting a OSError: The parameter is incorrect after calling await client.write_gatt_char(MY_UUID, msg). Is this related to this issue, or am I doing something wrong?

I have tried it on v0.13, v0.14, and the develop branch (v0.15.1 I believe)

Context:

async def run(address, name):
    print("Connecting to {} ({})...".format(name, address))
    async with BleakClient(address) as client:
        print("Connected")
        msg = initMsg()
        await client.write_gatt_char(MY_UUID, msg)
        print("Message sent")

...

loop = asyncio.get_event_loop()
loop.run_until_complete(
    run(address, name))

Jan 14 '22 01:01 kaedenbrinkman

The message needs to be a bytes-like object (Python buffer protocol). You didn't include the code for initMsg(), so we can't see what the problem is.

Jan 14 '22 01:01 dlech

initMsg() returns a byte array from a protobuf message: return my_proto_msg.SerializeToString()

When I do print(type(msg)), I see <class 'bytearray'>. Let me know if this still isn't enough information.