status() Shutting Off Power on v3.1 Device

[arrmo]

trafficstars

OK, I thought I was crazy before (in fairness, I likely was 😆), but now seeing it a second time. Let me explain ...

1. Had an earlier Tuya switch, it was v3.1 - and it seemed like when I did a status() query, it was (sometimes?) shutting the switch off. Wasn't 100% sure, but got Tuya to push a firmware upgrade ... it went to v3.3, and the issue went away. Case closed ... I thought, but then ...
2. Bought another switch, it's on v3.1 - and Tuya says there is no firmware update for this one. It does the same ... shuts off (often / always - to be confirmed yet) when I run a status() command.

So, time to figure out why? Is there a defined format for messages in and out? I can try to capture the traffic, and we can decipher then (i.e. tcpdump and/or Wireshark).

Thanks!

[jasonacox]

Hi Russell, you have some of the best problems. :).

I've heard others who have seen this with other 3.1 devices, even when not using a valid LOCAL_KEY. I have never seen it myself so hard to troubleshoot. It always goes away with upgrade so I suspect the websocket implementation for those particular devices is buggy and when we poll it direct, it has a chance of crashing in a way that causes a power cycle. There should be no way to hijack a Tuya switch without the LOCAL_KEY yet these devices seem easily vulnerable to attack without a key. My $0.02 is that I wouldn't run these on my local network without a firmware upgrade regardless of what I have it controlling (think Stuxnet type attack that could destroy whatever it is powering).

Having said that, I still would love to know if there is a way to identify these types of devices and if there is a custom way to manage communication with them without crashing. It is likely not high on most people's list since Tuya devices do not use local control (the Tuya device and the related SmartLife app communicate with the cloud, not the device directly, as you know). Those of us wanting local Tuya control are the only ones who would care. ;-)

Did you recently buy this device? If so, can you send me a link? If not too expensive, it may be a fun to dissect.

[arrmo]

Hi Russell, you have some of the best problems. :).

I'm not sure what that means, but my wife says the same 🤣.

wouldn't run these on my local network without a firmware upgrade

Understand - but I reached out to Tuya, they said this is the current firmware for this device. I thought about returning in then (just bought it), but the engineer in me said ... "no, I want to figure this out" 😄.

Having said that, I still would love to know if there is a way to identify these types of devices and if there is a custom way to manage communication with them without crashing. It is likely not high on most people's list since Tuya devices do not use local control (the Tuya device and the related SmartLife app communicate with the cloud, not the device directly, as you know). Those of us wanting local Tuya control are the only ones who would care. ;-)

Agreed! We are 100% aligned there. And I have been doing some debugging, a few thoughts / odd findings,

1. I turned on tcpdump, on my firewall, filtered traffic to this device. Found that the communication to the cloud is MQTT based (was a new one to me, bit of digging on that one). So,
2. Set up an MQTT server here (mosquitto), and https://github.com/TheAgentK/tuya-mqtt. Interestingly enough, I only seem to be able to make it switch off with MQTT if I sent the get-states command directly to a dps value (i.e. 19). Otherwise, through MQTT I can't seem to get it to turn off (in error, I can toggle it if desired). But then, I tried,
3. https://github.com/codeclinic/TuyaPower2MQTT/tree/master/tuyapower2mqtt ... and I again saw (MQTT) queries turning the device off. Hmm ... only with python code? An issue in the interface (in python) perhaps? Not sure, still digging.

Sort of clear? It is interesting, and I can use tuya-mqtt to subscribe / publish (communicate via MQTT), seems stable, no shut off.

Did you recently buy this device? If so, can you send me a link? If not too expensive, it may be a fun to dissect.

I did! As above, I just can't let it go now, the engineer has a hold of me 😆. Here is the link, https://www.amazon.com/gp/product/B07FDLBGQT/ref=ppx_yo_dt_b_asin_title_o03_s00?ie=UTF8&psc=1

Thoughts? Do you use MQTT under the hood? I just haven't had a chance to look yet.

[arrmo]

FYI, as a test (same switch), running MQTT, in a loop (same timing as tinytuya). So far, ~ 20 runs in, no shut off of the switch. So something in the command being sent? Will let this run overnight, let's see!

[arrmo]

OK, ran MQTT for 24 hours, query once a minute - no issues at all. So now to figure out what in the message is causing grief. Let me try to run some tcpdump captures, will let you know what I find.

[arrmo]

Hmmm ... OK, captured tcpdump from the internet to my switch => looks like all traffic is MQTT. Is that expected.

BTW, MQTT does respond quite quickly, and actually sends asynchronous updates, when dps values change.

[jasonacox]

I won't have time to dig into this until this weekend but I want to check the code you are using. Quick glance, it looks like it is a MQTT connector that uses tuyapi to talk to the Tuya device. I'm assuming:

MQTT Client/CLI -------> MQTT Server <------ (tuya-mqtt) ======> Device

Where

• --------> is MQTT traffic and
• ======> is Tuya protocol traffic

If you are seeing constant DPS changes, it is likely making a persistent connection to the Device (e.g. d.set_socketPersistent(True) with tinytuya).

[arrmo]

No worries! Just keeping you in the loop as I dig 😄. Your "diagram" looks right to me, but still digging.

Interestingly, I think (just happened) ... directly using tuyapi causes shutoff, but not with MQTT. Hmm ... need to noodle on that one a bit yet 🤣.

[arrmo]

OK, some digging with tcpdump and WireShark. I think (may have this wrong, but ...) that from the internet to the switch ... yep, it's MQTT (still not quite sure how WireShark is detecting that, I can't find a defined packet header anywhere). v3.1 is somewhat readable (or at least it makes sense), v3.3 is TLSv1.2 encrypted.

So that aside, looking locally - and using mosquitto + tuya-mqtt => I can query all day long, no switch toggling. I can also see the (TCP) packets, they look "similar" to what tinytuya sends. Do you have any info on the expected packet format? Then I can check if I see any packet related issues. Again, no panic - just trying to help.

Thanks!

[arrmo]

Oh, and one little tidbit - just in case you find it interesting, I did at least ... 🤣. With MQTT, if you are subscribed, when measurements change (e.g. power consumption), the switch (device) sends a message autonomously - kinda cool.

[arrmo]

FYI, looping right now, using tuyapi, getting dps info - not seeing the switch turn off so far (~ 20 loops complete, would have expected it). I do see some (slight?) differences in the packets sent across (no mqtt involved here, just tuyapi vs. tinytuya), thinking that's likely the delta. No panic, but when you have a chance - if you can just point me at the "packet builder", I can try to figure out if the deltas there are the issue.

I'm not looking at the TCP or IP headers, assuming this is just in the TCP payload (but feel free to disagree).

[jasonacox]

@arrmo Can you share the code you are using for both tuyapi and tinytuya? I would like to look deeper into the tuyapi code to see what we may be missing/different in tinytuya. I need to also revisit the localtuya work to see if there is any advancements there we could pull in.

As for packet payload, it depends on the command sent to the device. Check out the code and description here: https://github.com/jasonacox/tinytuya/blob/1f4644c78682f0cfded21b7b38fa9fee04341501/tinytuya/init.py#L217-L269

The payload is built by the generate_payload() function here: https://github.com/jasonacox/tinytuya/blob/1f4644c78682f0cfded21b7b38fa9fee04341501/tinytuya/init.py#L385-L456

[arrmo]

Can you share the code you are using for both tuyapi and tinytuya?

Sure, NP! Here is what I'm doing,

1. Python, tinytuya (dps results OK, but switch turns off) - tried it manually, code pasted here.

import tinytuya
d=tinytuya.OutletDevice('###', '192.168.2.159', '###')
d.set_version(3.1)
d.status()

2. JS / NodeJS, tuyapi (ran this in a loop, got sidetracked ... 18k iterations, 😆. No issues seen).

const TuyAPI = require('tuyapi');
const device = new TuyAPI({
  id: '###',
  key: '###',
  ip: '192.168.2.159',
  issueGetOnConnect: false});
(async () => {
  await device.connect();
  let status = await device.get({schema: true});
  console.log(`Current status: ${status.dps['1']}, power: ${status.dps['5']}`);
  device.disconnect();
})();

Make sense?

Check out the code and description here

Thanks! Seems I have been looking the right place. A couple notes, a. DP_QUERY seems to be missing dps {}, vs. tuyapi. Not sure it's critical, and I tried adding it locally - that's not (all of) it. From WireShark, I belive the prefix and suffix are being added correctly. b. The crc approaches look different - may be the same, haven't had a chance to dig more on that yet.

[arrmo]

OK, now this is really screwy 😜. Pulling my hair out, and I can't afford that! Let me explain what I tried,

1. Using WireShark, I captured one of the packets sent by tuyapi (NodeJS), wrote it to a binary file.
2. Used python to send that (raw) binary packet to the switch. I wasn't sure if it would work, as time is part of the information sent, but it seems to be OK => I get a sane (and correct) response back. So I have a "proven" good packet.
3. Put the python call (raw packet send, response) in a loop ... boom! After ~ a half dozen or so (it does vary, checked that), the switch turns off. I also tried changing the timing (~ every 15 sec, changed to ~ 5 sec ... still about the same number of send / receive operations to make it turn off). I also (SmartLife app) turned the switch back on, then restarted Python ... the output is correct, for another group of runs (< 10 each time).

Clear as mud? Is this somehow Python socket related? Or am I just nuts (entirely possible!). That packet format was OK with NodeJS for ~ 18k packets, so something happens when using Python?

Thanks!

[jasonacox]

Thanks for putting the code snips there! So there is one big difference. The tuyapi code is using persistent sockets and your tinytuya code is not. Make a slight change to see if this makes a difference:

import tinytuya
d=tinytuya.OutletDevice('###', '192.168.2.159', '###')
d.set_version(3.1)
# Keep socket connection open between commands
d.set_socketPersistent(True) 

By default tinytuya builds and tears down socket connections with each call. It would be interesting to see if this is what is causing the instability. Likely that would mean that the tuya device is keeping those connection threads alives and is hitting some resource limit causing a crash (power off?). One other tune we could make without changing code:

# Disable fast no-delay for TCP connect
d.set_socketNODELAY(False) 

Let me know what you find out. ;)

[arrmo]

Sure, will do! But one question - my (~ 18k) loops run as below ... so even NodeJS is tearing down every time?

#!/bin/bash
while :
do
        echo "$(date) Sending tuyapi command ..."
        node tuyapi.js
        sleep 15
done

Meaning - the loop is bash, calling node (JS file), not a loop in JS.

Thanks!

[arrmo]

You may be on to it! I ran 50 loops, persistence = True, and it stayed up. But then (to test), set it to False, Very quickly,

1: Sun Dec 20 02:55:16 PM CST 2020 Sending tinytuya command ...
{'1': True, '2': 0, '4': 0, '5': 0, '6': 1211, '7': 1, '8': 1739, '9': 17771, '10': 52153, '11': 395}
2: Sun Dec 20 02:55:21 PM CST 2020 Sending tinytuya command ...
{'1': True, '2': 0, '4': 0, '5': 0, '6': 1211, '7': 1, '8': 1739, '9': 17771, '10': 52153, '11': 395}
3: Sun Dec 20 02:55:26 PM CST 2020 Sending tinytuya command ...
Unexpected status() payload=b'json obj data unvalid'
Traceback (most recent call last):
  File "./tinytest.py", line 8, in <module>
    print(result['dps'])
TypeError: byte indices must be integers or slices, not str
4: Sun Dec 20 02:55:32 PM CST 2020 Sending tinytuya command ...
{'1': False, '2': 0, '4': 0, '5': 0, '6': 1216, '7': 1, '8': 1739, '9': 17771, '10': 52153, '11': 395}

Let me run it a bit longer, with persistence = True. And realizing - this may be persistence on the (Tuya switch) side? Just trying to understand.

Thanks!

[arrmo]

OK, 5k runs, no issue - switch still on. That's it, thanks!

I have to ask though ... why? 🤣.

Appreciate it!

[jasonacox]

From a code logic perspective, the only difference is that instead of immediately closing the connection after the payload is delivered by the device, the persistence logic leaves it open until python terminates. That would mean that the connection stays open slightly longer. Why would that help?

I love trying to solve puzzles like this. But at the same time, I am slightly worried about this device. I suspect you could run the same status query loop (non persistent) without a device key and it would eventually power off. But either way, even if there is some python socket or tinytuya protocol usage error here, there seems something vulnerable that is related to poor error handling on the device itself. All that to say, I wouldn't put this device on anything critical like a freezer. :)

I've been analyzing the tuyapi code and there are some things I like and want to try to incorporate. For one thing, the persistence mode is default. However, I notice that when I leave persistence on for tinytuya, subsequent d.status() calls are getting bloated payloads that have the right data but no longer fit within the packet size (it adds what appears to be extraneous/extended data). The tuyapi code handles this by extracting the payload between the prefix and postfix signatures and recursively processing the "remainder" in case there are other payloads there. I think that is the right approach so I intend to update the parsing in tinytuya to do something similar.

Thanks for all your investigation and help with this!

[arrmo]

That would mean that the connection stays open slightly longer. Why would that help?

Agreed! And was thinking the same thing ... 😆. Wondering if it gives the Tuya device time to clean up on it's side perhaps? I did accidentally leave it running, and it completed 10k runs without an issue. A bit better than < 10 😜. So clearly the issue, just not 100% sure why it matters (except clean up perhaps).

All that to say, I wouldn't put this device on anything critical like a freezer. :)

Agreed! And it doesn't occur with v3.3 devices, only v3.1. Odd.

subsequent d.status() calls are getting bloated payloads that have the right data but no longer fit within the packet size (it adds what appears to be extraneous/extended data).

Really? I don't see that. I accidentally left WireShark on as well (above), captured almost 90k packets - and they were all a consistent size. You see the response packet size growing? With a v3.1 device?

I think that is the right approach so I intend to update the parsing in tinytuya to do something similar.

Agreed, makes a lot of sense. Just yell if you want me to try something here.

Thanks for all your investigation and help with this!

No worries at all, glad to help. Thanks so much for all you have done here, really is a great tool / driver. Appreciate it!

[arrmo]

Arrgh! OK, thought it was all good - then moved the switch to the "freezer" LOL. Actually, outdoor lights. Hmmm - still have persistence on, and it's switching off again. More than just persistence perhaps. Need to dig more.

[arrmo]

OK, I see the difference (moved to different code, vs. the test I was running) => seems even when calling the Python code (that accesses d.status()), a delay is needed after the call, before the script exists. For example (this is called by a script),

1. No delay, this breaks,

            device.set_socketPersistent(True)
            currData = device.status()

2. Delay, all good

            device.set_socketPersistent(True)
            currData = device.status()
            time.sleep(2.0)

In both cases, persistence is on. Do you see it acting similarly?

Thanks!

[jasonacox]

Interesting. Maybe whatever 3.1 firmware that device is using requires additional time to clean up the TCP handshake? Or it crashes? I wonder what it is about NodeJS socket handling that is different from python if the tuyapi version doesn't experience this. What OS are you using?

[arrmo]

Yep, wondering here as well! I'm using Ubuntu 20.10 (OS).

I set my code up to only add the delay for v3.1 devices. So far, all working OK.

Thanks!

[arrmo]

Hi,

Just curious - have you been able to duplicate this? No biggie!

Thanks!

[jasonacox]

Hi Russell, I haven't been able to duplicate this. Have you figured out the right "delay" amount for these 3.1 devices? I see 2s above but wondering if it could be an optional setting for 3.1 devices I could accomodate in the code. Also, have you tried with different python versions?

[arrmo]

Hi.

I haven't optimized it, but did this in my code - and works just fine. I likely could go less, but only have a couple devices, poll them once every 5 min - so not worth shaving 100 ms ... LOL.

            if currdevice['ver'] == 3.1:
                time.sleep(0.5)

And only have tried Python 3.

Thanks!