tinytuya icon indicating copy to clipboard operation
tinytuya copied to clipboard

TH sensors: json obj data unvalid

Open fkonrad opened this issue 11 months ago • 12 comments

Hi,

I'm trying to get temp, humidity out of a ZY-TH02 sensor locally, It's correctly configured with the smart app, I've got the localkey, but I'm not able to communicate with the sensor:

  • It broadcasts the "discovery" UDP packet for some time when I power it up (by placing the batteries) and then go idle.
Unknown v3.4 Device   Product ID = x3o8epevyeo3z3oa  [Valid Broadcast]:
    Address = 192.168.0.110   Device ID = bf76b1b98633be00c0zre6 (len:22)  Local Key =   Version = 3.4  Type = default, MAC = 
    No Stats for 192.168.0.110: DEVICE KEY required to poll for status
  • During that time I try to get the status with:
while(True):                                                                                                                                                                                                                                  
    d = tinytuya.OutletDevice('bf76b1b98633be00c0zre6', '192.168.0.110',  'xxx')                                                                                                                                                              
    d.set_version(3.4)                                                                                                                                                                                                                        
    data = d.status()                                                                                                                                                                                                                         
    print(data)                                                                                                                                                                                                                               
    data = d.status()                                                                                                                                                                                                                         
    print (data)

At the first round it get "json obj data unvalid" and switch to device22:

DEBUG:decrypted 3.x payload=b'json obj data unvalid'
DEBUG:payload type = <class 'bytes'>
DEBUG:'data unvalid' error detected: switching to dev_type 'device22'

The next rounds it keeps timeouting waiting for replies:

DEBUG:final payload: b'3.4\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00{"devId":"bf76b1b98633be00c0zre6","uid":"bf76b1b98633be00c0zre6","t":"1738028383","dps":{"1":null}}'
DEBUG:payload encrypted=b'000055aa000000060000000d000000a4e4c324393ba5b22ca63886763b3a7abfbd19a19f393870e47b686b2ccc5ab30ea373a7ac56258c6082d62d9ad46aa31106bae1c08d35739d35dde516429a204d06d0497c253bcb5f2f5134e6b0ca172a3828b10cceaee16823907cacfcf42f8f897a3124d6205d97887fa653d453e4255d965acdfccaf6314d1221e44cd7737d303036811af30f966af166d7dfb9dcd1a8ecccb18cbac1cf6064567545d962f40000aa55'
DEBUG:Timeout in _send_receive() - retry 1 / 5

Also it seems like that kind of device have been successfully used with LocalTuya here.

Do I miss anything here? Does somebody got any success with those devices?

Best Regards,

fkonrad avatar Jan 28 '25 01:01 fkonrad

You may want to try the https://github.com/jasonacox/tinytuya/blob/master/examples/monitor.py script.

Also, you can specify device22 on initialization:

a = tinytuya.OutletDevice('here_is_my_key', '192.168.x.x', 'secret_key_here', 'device22', version=3.4)
data =  a.status()
print(data)

You could also set up a constant ping to the device once it comes online to keep it awake.

Note: Battery powered devices are not compatible with local access. They are designed to send their data to the cloud and go to sleep. If you get a local access to work, you are likely going to drain the battery fairly quickly. But good luck on your reserach!

jasonacox avatar Jan 29 '25 04:01 jasonacox

Hi, ok thanks for your pointer.

What I had in mind actually was detecting the UDP broadcast and query the dps just after, (there is time to do it if the device doesn't have access to the tuya server.) Unfortunately that query packet isn't answered by the device, maybe the query is malformed or the BK7231N firmware has been programmed not to answer..

fkonrad avatar Jan 30 '25 11:01 fkonrad

I suspect the UDP broadcast it sent when it wakes up to send an update to the cloud. But that's a clever approach. Let us know if you figure it out!

jasonacox avatar Jan 31 '25 05:01 jasonacox

These things are cheap enough I went ahead and picked up a couple myself, they should be here later next week. Since they are responding with an 'unvalid' error they should be online long enough to poll them. Hopefully I can figure out what sort of magic sequence they're expecting.

uzlonewolf avatar Jan 31 '25 07:01 uzlonewolf

Did you manage to make it work? I have set up a monitoring script but it is not working properly. Most of the times it just gets the battery status but not temperature nor humidity. If I force the sensor to send data by taking out and in the batteries it usually has time to get all the data.

Marcosld avatar Jul 04 '25 10:07 Marcosld

I'm afraid, not on my side, I never managed to get anything from it and took an other path. Note that maybe if your device doesn't have access to the cloud it will have more time to answer your tuya requests.

fkonrad avatar Jul 04 '25 17:07 fkonrad

I tried blocking outcoming requests from the sensor but no luck. It looks like it stays online for the same amount of time. On the other hand the device sometimes answers to tinytuya and sometimes does not, and when it answers it is not guaranteed that it reports all the dps. Half of the times it just reports the battery level XD

Marcosld avatar Jul 04 '25 19:07 Marcosld

After my devices came in they kinda got thrown into a corner and forgotten about 😅 The ping from this thread reminded me to look into them.

Looking at packet captures, when the device wakes up it first sends a "Logical-Link Control, Supervisory, Receiver not ready" broadcast, then 2 ARP requests for its own IP address, and then starts communicating. Unfortunately it never sends a Tuya device broadcast. While it would be technically possible to listen for the receiver-not-ready or ARP broadcasts, doing so would require using raw sockets and thus running with Admin privileges.

To avoid that, I wrote a script that simply spams TCP connection requests until it gets a response. It's not pretty, but so far seems to be working reasonably well. I'll need to play around with it a bit to see how much we can back off on the TCP SYN spamming and still get reliable connections. Edit: backed off the spamming a bit, it now waits 1s between attempts, making the retry time 1 + 0.1 = 1.1s.

import time
from datetime import datetime
import tinytuya

#tinytuya.set_debug()

d = tinytuya.Device( '...', '...', local_key='...', version=3.4, persist=True, connection_timeout=1, connection_retry_limit=999, connection_retry_delay=0.1 )
d.disabledetect = True
#d.add_dps_to_request( (27,46,101) )

start_time = data_time = 0

while True:
    # FIXME: find a better way to detect if a device is online
    sock = None
    d.set_socketTimeout( 1 )
    while sock is not True:
        sock = d._get_socket(True)
        print( 'got sock?', sock )

    # device is now online!
    print( datetime.now(), time.time() )
    print( 'time since last connect:', time.time() - start_time )
    print( 'time since last data:', time.time() - data_time )
    start_time = time.time()
    d.set_socketTimeout( 30 ) # give it up to 30 seconds to start sending data
    d._have_status = True # allow cached_status() even though device does not support status
    data = d.receive()
    print( 'data:', data, 'at', time.time() )
    d.set_socketTimeout( 1 ) # once it starts sending data, transmission will be finished after 1s
    got_data = False
    while data is not None:
        data_time = time.time()
        got_data = True
        data = d.receive()
        print( 'data:', data, 'at', time.time() )
    print( 'status:', d.cached_status( nowait=True ) )
    if got_data:
        # give the device some time to go back to sleep
        time.sleep(30)
    print( '' )

Results in:

got sock? True
2025-07-05 03:58:15.875922 1751713095.8759258
time since last connect: x
time since last data: x
data: {'protocol': 4, 't': 7, 'data': {'dps': {'27': 245}}, 'dps': {'27': 245}} at 1751713098.1257625
data: {'protocol': 4, 't': 7, 'data': {'dps': {'46': 52}}, 'dps': {'46': 52}} at 1751713098.34971
data: {'protocol': 4, 't': 7, 'data': {'dps': {'101': 'middle'}}, 'dps': {'101': 'middle'}} at 1751713098.6039958
data: None at 1751713103.6091821
status: {'protocol': 4, 't': 7, 'data': {'dps': {'27': 245, '46': 52, '101': 'middle'}}, 'dps': {'27': 245, '46': 52, '101': 'middle'}}

got sock? True
2025-07-05 04:00:21.765876 1751713221.7658808
time since last connect: 125.88994669914246
time since last data: 123.16188788414001
data: {'protocol': 4, 't': 3, 'data': {'dps': {'27': 270}}, 'dps': {'27': 270}} at 1751713223.4440668
data: {'protocol': 4, 't': 3, 'data': {'dps': {'46': 44}}, 'dps': {'46': 44}} at 1751713223.6485229
data: {'protocol': 4, 't': 4, 'data': {'dps': {'101': 'middle'}}, 'dps': {'101': 'middle'}} at 1751713223.852593
data: None at 1751713228.8576763
status: {'protocol': 4, 't': 4, 'data': {'dps': {'27': 270, '46': 44, '101': 'middle'}}, 'dps': {'27': 270, '46': 44, '101': 'middle'}}

got sock? True
2025-07-05 04:02:40.122693 1751713360.1226974
time since last connect: 138.35680747032166
time since last data: 136.2700960636139
data: {'protocol': 4, 't': 12, 'data': {'dps': {'27': 256}}, 'dps': {'27': 256}} at 1751713361.8501363
data: {'protocol': 4, 't': 12, 'data': {'dps': {'46': 44}}, 'dps': {'46': 44}} at 1751713362.1060185
data: {'protocol': 4, 't': 13, 'data': {'dps': {'101': 'middle'}}, 'dps': {'101': 'middle'}} at 1751713362.3093226
data: None at 1751713367.3143103
status: {'protocol': 4, 't': 13, 'data': {'dps': {'27': 256, '46': 44, '101': 'middle'}}, 'dps': {'27': 256, '46': 44, '101': 'middle'}}

got sock? True
2025-07-05 04:04:35.432131 1751713475.4321358
time since last connect: 115.30943417549133
time since last data: 113.12281966209412
data: {'protocol': 4, 't': 6, 'data': {'dps': {'27': 250}}, 'dps': {'27': 250}} at 1751713477.1990037
data: {'protocol': 4, 't': 6, 'data': {'dps': {'46': 45}}, 'dps': {'46': 45}} at 1751713477.4591484
data: {'protocol': 4, 't': 7, 'data': {'dps': {'101': 'middle'}}, 'dps': {'101': 'middle'}} at 1751713477.749999
data: None at 1751713482.7550724
status: {'protocol': 4, 't': 7, 'data': {'dps': {'27': 250, '46': 45, '101': 'middle'}}, 'dps': {'27': 250, '46': 45, '101': 'middle'}}

got sock? 905
got sock? True
2025-07-05 04:13:32.602787 1751714012.6027925
time since last connect: 537.1706528663635
time since last data: 534.8528017997742
data: {'protocol': 4, 't': 3, 'data': {'dps': {'27': 246}}, 'dps': {'27': 246}} at 1751714014.0121326
data: {'protocol': 4, 't': 3, 'data': {'dps': {'46': 46}}, 'dps': {'46': 46}} at 1751714014.2559116
data: {'protocol': 4, 't': 3, 'data': {'dps': {'101': 'middle'}}, 'dps': {'101': 'middle'}} at 1751714014.4826272
data: None at 1751714019.4877315
status: {'protocol': 4, 't': 3, 'data': {'dps': {'27': 246, '46': 46, '101': 'middle'}}, 'dps': {'27': 246, '46': 46, '101': 'middle'}}

got sock? 905
got sock? 905
got sock? 905
got sock? 905
got sock? 905
got sock? 905
got sock? 905
got sock? True
2025-07-05 04:57:17.643254 1751716637.6432583
time since last connect: 2625.040449142456
time since last data: 2623.160612344742
data: {'protocol': 4, 't': 3, 'data': {'dps': {'27': 241}}, 'dps': {'27': 241}} at 1751716639.0453358
data: {'protocol': 4, 't': 3, 'data': {'dps': {'46': 48}}, 'dps': {'46': 48}} at 1751716639.2830324
data: {'protocol': 4, 't': 3, 'data': {'dps': {'101': 'middle'}}, 'dps': {'101': 'middle'}} at 1751716639.54998
data: None at 1751716644.5542982
status: {'protocol': 4, 't': 3, 'data': {'dps': {'27': 241, '46': 48, '101': 'middle'}}, 'dps': {'27': 241, '46': 48, '101': 'middle'}}

uzlonewolf avatar Jul 05 '25 11:07 uzlonewolf

I forgot to add, they do not seem to support status() at all, they only broadcast async DP updates. The WiFi version on my modules is v2.1.8, same as the problematic TH01 sensor in #597.

uzlonewolf avatar Jul 05 '25 11:07 uzlonewolf

I used your code and it seems to be working fine! thanks! perhaps I'll try to investigate into capturing the packages as I am running this in a docker container to send data to home assistant and I don't mind running it with admin privileges.

Marcosld avatar Jul 15 '25 18:07 Marcosld

Not to pollute the thread but I have a smart curtain that lost all ability to be controlled after a power outage. I tried power cycling and deleting and re-pairing in the app but tinytuya scan - debug kept giving me:

Smart curtain   Product ID = owflx7mv9vxb0mct  [Valid Broadcast]:
    Address = 192.168.1.80   Device ID = ***pfgu (len:22)  Local Key =***  Version = 3.4  Type = default, MAC = ***:90:95
    Polling 192.168.1.80 Failed: No response

tinytuya also showed:

DEBUG:Received valid UDP packet: {'ip': '192.168.1.80', 'gwId': '***pfgu', 'active': 2, 'ablilty': 0, 'encrypt': True, 'productKey': 'owflx7mv9vxb0mct', 'version': '3.4', 'token': True, 'wf_cfg': True}

but not any more information about the status of the device. I don't know what 'token' or 'wf_cfg' are about.

My "Main module" is version 2.7.7 and my "MCU module" is 1.0.0. It's a WBR3 chip.

I tried running the above code and then changing the state of the curtain using the app, and everything is working again!

PS C:\Users\o> & C:/Python312/python.exe c:/Users/o/testtuyasock.py
got sock? True
2025-09-09 09:56:37.284661 1757433397.284662
time since last connect: 1757433397.284662
time since last data: 1757433397.284662
data: {'protocol': 4, 't': 1757433418, 'data': {'dps': {'1': 'open'}}, 'dps': {'1': 'open'}} at 1757433418.661988
data: {'protocol': 4, 't': 1757433418, 'data': {'dps': {'2': 50}}, 'dps': {'2': 50}} at 1757433418.716008
data: None at 1757433419.7284288
status: None

I don't really know what specifically I did or what is necessary to get the device to work again, but perhaps some combination of monitoring using the above code and sending commands causes the device to start sending data locally? I should also note that I extended my Tuya IOT Core subscription while fiddling with all of this. None of my 10 or so other Tuya devices were experiencing these troubles despite my Tuya IOT Core subscription lapsing so I'm not sure if that's related either.

I don't really have enough information to be sure what aspect of my fiddling resulted in the device working again but the device started working while I was messing around here so maybe there's something here for other people experiencing issues with version 3.4 devices that just stop working.

backcountrymountains avatar Sep 09 '25 16:09 backcountrymountains

@backcountrymountains While I do not have a curtain controller to test this with, that behavior looks identical to most IR blasters - after a power cycle they do not return any response to .status() until a command is sent. This is a quirk of the device itself and there's nothing we can do about it in TinyTuya. We should probably update the documentation to note this behavior though...

uzlonewolf avatar Sep 10 '25 13:09 uzlonewolf