Download historical data from device

[wcbonner]

I'd have to learn the communication protocol completely from scratch.

[wcbonner]

I've got a btsnoop log of the data from the official app and decoding the data itself seems relatively straightforward.

I'm still working to understand how to connect and initiate the transfer.

[sebest]

Did you do any progress on this front?

[wcbonner]

Did you do any progress on this front?

I've made no process on historical data download.

I learned enough to recognize that it requires getting the device in direct communication mode and that seems to cause more battery drain as well as blocking advertisements during that time.

The bigger issue was more that I didn't know the correct sequence of Bluetooth commands, let alone the communication sequence for these devices.

[sebest]

Using your bluetooth data dump and wireshark I was able to figure out how to download the historical data (below is a python code to do it).

In a nutshell, I subscribe to 494e5445-4c4c-495f-524f-434b535f2013 and then I write 33017081000100000000000000000000000000c2 to 494e5445-4c4c-495f-524f-434b535f2012.

That triggers receive the notification that you described in your analysis with notifications containing 6 samples. the first 4 bytes are an integer value for the index of the samples. The value decreases by 6 at each notification. The first index is 28800, the next one is 28794 ... I often only receive results until index ~13032 I think that the number of notification that will receive is not consistent (It around 15k samples received) and I was not able to figure out how to request a specific number of samples (I don't even know if it is possible).

I took notes about the other values that are sent to UUIDs ending by 2012 and 2011, there are not than many different values written to the characteristics from your dump and I was not able to figure out a pattern yet.

Also, I often have issues connecting to the device, it requires multiple retries.

import asyncio
from bleak import BleakClient

addresses = [
    # "A4:C1:38:F7:A9:57",
    "A4:C1:38:AC:A8:AF",
    # "A4:C1:38:33:C8:C9",
]


def print_values(data):
    offset = 4
    # This is the sample idx, this value decrease by 6 at each notification
    print(int(data.hex()[0:offset], 16))

    # We have 6 samples per notication
    for _ in range(6):
        s = data.hex()[offset : offset + 6]
        offset += 6
        temp_humid = int(s, 16)
        print(
            "temp=%s humid=%s"
            % (
                temp_humid / 10000,
                temp_humid % 1000 / 10,
            )
        )


async def run():
    while True:
        for address in addresses:
            print(f"Connecting to {address}")
            client = BleakClient(address, timeout=30, adapter="hci1")
            try:
                res = await client.connect()

                def callback(sender: int, data: bytearray):
                    print_values(data)

                # 494e5445-4c4c-495f-524f-434b535f2013 (Handle: 24): Unknown
                await client.start_notify(24, callback)

                # 494e5445-4c4c-495f-524f-434b535f2012 (Handle: 20): Unknown
                res = await client.write_gatt_char(
                    20,
                    bytes.fromhex("33017081000100000000000000000000000000c2"),  # 2286
                )

                await asyncio.sleep(60)
                await client.stop_notify(24)

            except Exception as e:
                print(e)
            finally:
                await client.disconnect()
            await asyncio.sleep(0.1)


loop = asyncio.get_event_loop()
loop.run_until_complete(run())

[wcbonner]

Thanks for the python code.

I remember the issue with having to attempt to connect multiple times, which was part of the reason I set the project aside.

I know that even the Govee App often takes multiple refresh attempts to get the data to start downloading. I also know from watching it, that when it requests data, it has a way of only retrieving the most recent data instead of all of the data, which is apparent by how long the transfer takes.

Do you know if your use of handle 24 is consistent from device to device?

[sebest]

I also know from watching it, that when it requests data, it has a way of only retrieving the most recent data instead of all of the data, which is apparent by how long the transfer takes.

I also noticed this, but I was not able to understand yet how to do it from the data dump.

Do you know if your use of handle 24 is consistent from device to device?

I have 3 identical devices (GVH5072) and they all work with handle 24.

I'm new to bluetooth and did not understand yet what is the link between handle and UUID, but what I noticed from your dump is it seems that you have 2 different device types.

I both case the characteristic UUID to write to is 494e5445-4c4c-495f-524f-434b535f2012 and the value to write is the same, BUT the handle as reported in wireshark is different. Same thing for reading the data. (see the screenshot below) So using UUID works with different devices as far as I can tell.

[shuckc]

Hi @sebest thanks for your python code - most useful.

I used bleak to write a device scan and announcement decoding for a couple of the new devices H5174 and H5179 see https://github.com/shuckc/govee_logger/blob/main/scan.py . I am now looking at downloading historical data. I examined the BT packet captures in this repository and they broadly make sense. I found my my devices use completely different UUIDs though.

I have the Govee app for iOS, so I installed an Apple Bluetooth debugging profile on my device, then ran the "PacketLogger" tool which is part of "Additional Tools for Xcode". With PacketLogger I started a recording connected over USB, then opened up the Govee iOS app and did the synchronization process for both devices - one was a short download of maybe 8 hours, for the other device it was several days behind.

One thing I noticed is that the app gets the most recent temperatures first - if you abandon the sync process, then re-open it you can see it has filled in more recent missing data first.

Whilst the recordings seem to be an Apple specific capture format I'll see if I can get them shared. I haven't gone through them yet to find out how different the download process is from your devices but hope to do so soon.

[wcbonner]

@shuckc

I used bleak to write a device scan and announcement decoding for a couple of the new devices H5174 and H5179 see https://github.com/shuckc/govee_logger/blob/main/scan.py . I am now looking at downloading historical data. I examined the BT packet captures in this repository and they broadly make sense. I found my my devices use completely different UUIDs though.

Cool to see more devices using similar but different data structures. I wonder how different the H5174 is from the H5074 beyond the use of three AAA batteries instead of one CR2477. I don't have a need for more devices in my home, but I'd kind of like to add support in my code. Would you be able to post the hex content of a couple of advertising messages from each of the devices you've decoded? (H5174, H5179)

Thanks again. Wim

[shuckc]

I found these 4-byte "index values" (at least on 5179's) are the current UNIX timestamp in seconds, divided by 60. So historical data is at minute resolution. I have mostly decoded the protocol over the xx2011 channel and I was expecting to see a clock query/synchronisation step, but so far I can't find it. The xx2011 protocol has a simple XOR checksum and lets you query firmware and hardware versions.

I agree with your findings for xx2012 (request) and xx2013 bulk data download UUIDs - they are unchanged. I was thrown by the devices offering different service UUIDs which seem to be something else entirely.

[shuckc]

The final byte on your request packet above looks to be the same checksum scheme used on 5179's UUID xx2011, ie. byte-wise XOR over the whole message.

Requesting historical data on the H5179 seems a bit different: send the start and end index values on xx2012 prefixed by two bytes of zeros, then wait for the bulk data, which arrives in groups of 3 or 4 readings.

Historical data for the H5174 matches what you have above - requested by 0x3301, payload 2-byte indexed and arrives in groups of 6 readings.

Both devices use the same 'set clock' message of 0x3310 with payload of (UNIX timestamp / 60)

I've documented both protocols at https://github.com/shuckc/govee_logger/#readme

[sebest]

@shuckc

Historical data for the H5174 matches what you have above - requested by 0x3301, payload 2-byte indexed

Were you able to figure out the format of the request for bulk download on H5174, from what I understand it starts by 0x3301 then followed by the first index, but I'm not sure about the remaining part.

Here is the request that you had in your logs:

0x3301 1c9b 0000 0000 0000 0000 0000 0000 0000 00B5

Here is one I had in mine:

x3301 7081 0001 0000 0000 0000 0000 0000 0000 00c2

1c9b and 7081 are the starting indexes, but mine is followed by 0001

Is the last byte a checksum? how is it generated?

Thank you

[sebest]

Is the last byte a checksum? how is it generated?

Nevermind, you mentioned it was a byte wise xor of the message

[shuckc]

I think the 0001 or 0000 is the ending index. The difference is if the app has received an announcement in the last 60 seconds, if so it does not need the most recent value.

It looks like stitching together by index must occasionally go wrong if the minute changes while the requests and responses are in flight.

The newer protocol putting 4-byte minute timestamps on the data is more robust to this sort of thing. I have no idea how to get the starting index yet, there does not seem to be a message exchange to find the earliest. I might need to wipe the app and restart it under the bluetooth logger to see how they handle no local history.

0x3301 7081 0001 0000 0000 0000 0000 0000 0000 00c2 checksum (last byte) is 33^01^70^81^01 = 0xC2.
(I've omitted the ^00 for the remaining bytes as they don't change the result)

[sebest]

I wrote this function to generate the payload to download the data on GVH5072 (which seems identical to your H5174):

# Values are returned from the oldest (max 28800) to the newest (min 0).
def get_index_range(begin=0, end=0):
    idx_max = max(begin, end)
    idx_min = min(begin, end)
    assert idx_max <= 28800, "max index for range is 28800"
    assert idx_min >= 0, "min index for range is 0"

    index_range = struct.pack(">hhh", 13057, idx_max, idx_min)

    checksum = 0
    for b in index_range:
        checksum ^= b

    index_range += checksum.to_bytes(14, "big")

    return index_range

It looks like it can store up to 28800 values, each value represent 1 minute. So it is about 20 days of history.

[shuckc]

Handy! I think you are right about the byte order of the start index, which is different to other messages I have written.

I added some rx/tx checksum calculations yesterday and now I've written code for downloading from H5174 based on yours and found a few things in the result.

I first tried a range of 28,800 - 0 however the data stream stopped arriving at index 1,745. When I repeated range ~11 minutes later of 28,800 - 0 the download stopped at index 1,800?? I drop the range to 20,800 - 0 the download completes OK.

 Govee H5174 A4:C1:38:86:6B:E0 waiting for bulk data from 20800 to 0
 20800 t0=178492 t1=178493 t2=178493 t3=178493 t4=178494 t5=178494
 20794 t0=178494 t1=178494 t2=178494 t3=177494 t4=177495 t5=177495
 20788 t0=177495 t1=177495 t2=176495 t3=176496 t4=176496 t5=176496
 20782 t0=176496 t1=176497 t2=176497 t3=175497 t4=175497 t5=175497
 20776 t0=175497 t1=175497 t2=175498 t3=174498 t4=174498 t5=174499

    28 t0=198407 t1=201548 t2=202408 t3=201405 t4=199407 t5=198412
    22 t0=196413 t1=195414 t2=194416 t3=194417 t4=194420 t5=193421
    16 t0=193422 t1=192423 t2=192424 t3=192426 t4=191427 t5=191426
    10 t0=191428 t1=191430 t2=190428 t3=190431 t4=190431 t5=189432
     4 t0=189431 t1=189433 t2=189433 t3=189434 t4=189434 t5=16777215

The final row suggests the most recent data value is on the right hand side?

When I line up the data from a range, the samples 'move back' through the index as time advances. ie. I think the basic idea of the index being "number of minutes ago" is completely correct.

I also found a rare-ish case where only one value is returned in the result (index only moves by one) and the 'missing' values are replaced by 0xFFFFFF:

 index 3-byte temp/humidity values (humidity=t%1000, temp=t/1000)
 20838 t0=190473 t1=189473 t2=189474 t3=189474 t4=189474 t5=189475
 20832 t0=189475 t1=188475 t2=188476 t3=188476 t4=188476 t5=188477
 20826 t0=188477 t1=188477 t2=187477 t3=187478 t4=187478 t5=186479
 20820 t0=186479 t1=186479 t2=186479 t3=186480 t4=186480 t5=185480
 20814 t0=185481 t1=16777215 t2=16777215 t3=16777215 t4=16777215 t5=16777215  <= values skipped
 20813 t0=185481 t1=185481 t2=184481 t3=184482 t4=184482 t5=184482
 20807 t0=184483 t1=184483 t2=183483 t3=183484 t4=183484 t5=183486
 20801 t0=183487 t1=183486 t2=183488 t3=183487 t4=182487 t5=182488
 20795 t0=182488 t1=182488 t2=182488 t3=182488 t4=181488 t5=181488

See that the index changes from 20814 to 20813 rather than the usual 6 entries When I re-queried the range, the skipped values did not occur, ie. everything arrived in rows of 6.

20880 t0=196462 t1=195463 t2=195463 t3=195463 t4=195464 t5=194464
20874 t0=194464 t1=194465 t2=194465 t3=194466 t4=194466 t5=194467
20868 t0=194467 t1=193467 t2=193468 t3=193468 t4=193468 t5=193469
20862 t0=192469 t1=192469 t2=192470 t3=192470 t4=191470 t5=191471
20856 t0=191471 t1=191471 t2=191472 t3=190472 t4=190473 t5=190473
20850 t0=189473 t1=189474 t2=189474 t3=189474 t4=189475 t5=189475
20844 t0=188475 t1=188476 t2=188476 t3=188476 t4=188477 t5=188477
20838 t0=188477 t1=187477 t2=187478 t3=187478 t4=186479 t5=186479
20832 t0=186479 t1=186479 t2=186480 t3=186480 t4=185480 t5=185481
20826 t0=185481 t1=185481 t2=184481 t3=184482 t4=184482 t5=184482
20820 t0=184483 t1=184483 t2=183483 t3=183484 t4=183484 t5=183486
20814 t0=183487 t1=183486 t2=183488 t3=183487 t4=182487 t5=182488
20808 t0=182488 t1=182488 t2=182488 t3=182488 t4=181488 t5=181488
20802 t0=181488 t1=181489 t2=181489 t3=181489 t4=180489 t5=180490
20796 t0=180490 t1=180490 t2=179490 t3=179491 t4=179491 t5=179491
20790 t0=179491 t1=179491 t2=179492 t3=179492 t4=178492 t5=178493
20784 t0=178493 t1=178493 t2=178494 t3=178494 t4=178494 t5=178494
20778 t0=178494 t1=177494 t2=177495 t3=177495 t4=177495 t5=177495
20772 t0=176495 t1=176496 t2=176496 t3=176496 t4=176496 t5=176497
20766 t0=176497 t1=175497 t2=175497 t3=175497 t4=175497 t5=175497
20760 t0=175498 t1=174498 t2=174498 t3=174499 t4=174499 t5=174499
20754 t0=174499 t1=174499 t2=174499 t3=174500 t4=174500 t5=173500

Since this is transient I would guess it will be something like a FIFO buffer underflow between the memory on the device and the bluetooth transceiver/translator. Worth knowing about!

[shuckc]

I managed to capture the download started/complete messages on xx2012 channel too:

WR > 3301514000000000000000000000000000000023
VR < 3301000000000000000000000000000000000032 # Download accepted
then much later:
VR < ee010d8b00000000000000000000000000000069  # Downloaded complete

The 'download complete' message has the usual checksum but msg type ee01 isn't in reply to any message, and I have no idea where 0d8b comes from (have seen 04c5 before).

[wcbonner]

@sebest and @shuckc I thought I'd mention that I've finally spent the time learning a bit about GATT communication over Bluetooth LE, and there's now code that will download historical data from the devices I've got. It has been tested with the h5074, h5075, h5174, and h5177.

Right now it's hard coded to request all 20 days of data, but I hope to make it more flexible over the next couple of days and make it a little more sensible in normal operation.

[wcbonner]

I've figured out that I can safely request 0xffff from any of the devices I own and it will return the maximum available.

I point this out because it seems that while most of the devices keep 20 days worth of information, some contain a full month.

[wcbonner]

download appears to be working reliably now.