v6.0.0 Discussion

[codetheweb]

trafficstars

This issue will hopefully contain a more detailed plan and checklist in the future, but for now here's what we have

• [ ] Finalize interface
• [ ] Figure out general structure of TuyAPI
• [ ] Add tests
• [ ] Add documentation

Not sure how you want to split up the work @kueblc. I can also work on @tuyapi/stub or similar.

[kueblc]

Been a busy week, will follow up

[py60800]

Hi,

I don't know if this the right place to post the information but I want to let you know that I have developped a tool to dump communication with tuyadevices. It can be found here: https://github.com/py60800/tuyadump.

It is based upon gopacket, it can decrypt tuya commands (if provided with the keys!!).

By the way, you will find here a golang port of the API https://github.com/py60800/tuya. Still experimental...

[codetheweb]

That's cool @py60800, I'll add both projects to the README. :)

[kueblc]

Update; I've been focused on evaluating and reverse engineering the recent activity from Tuya, seems they are starting to take an active stance against local control and firmware freedom. :frowning_face:

Still a lot of work to be done, and I've been trying to get my hands on every firmware sample I can to speed up the process. If you come across any firmware bins or network captures of these recent updates please let me know!

[p3g4asus]

To give my contribute, If anyone is interested, I developed a python library that supports gocomma r9 that is a tuya smart remote. I developed also home-assistant remote component that supports it. I would like to thank you @codetheweb for all your hard work with Tuya devices and for your excellent tutorial on getting id and key out of the device.

[kueblc]

Some thoughts I've gathered over the past year or so:

1. Only need to get() on connect and reconnect, should never need to call this manually; proactive updates should keep everything up to date. Store this in this.dps.
2. If the above is implemented, we can make the user facing get() simply reference this.dps and move the current get() code into a private function, something like _update() or _dpQuery(). Or don't have get() at all and just expose this.dps, firing events to notify of updates. Or get() could be the "schema-tized" version along with direct access via this.dps.
3. MessageParser.parse() should return the return code. If non-zero, fire an error event, but attempt to process the rest of the queue.
4. Heartbeat should timeout if we don't get a response in a reasonable time. Throw an error or disconnect event or both.
5. Device discovery should be its own file/class. If we then assume TuyaDevice is always being called with the required ip id and key we can then remove some safety checks.
6. Perhaps encrypt should only take a string and only return a Buffer. If you need base64 encoding just call toString('base64') on the returned Buffer. This reduces logic in encrypt while improving readability where it is used, as it becomes abundantly clear when it is in base64 format.
7. Similarly with decrypt, it should only return a string. JSON decoding only needs to take place once, after decrypting. No need to attempt to JSON decode twice.
8. MessageParser could be delegated to only process the TuyaFrame, and leave payload processing to another version specific class (3.1, 3.3 ...). The TuyaFrame format is unchanging between versions. This will make a cleaner approach to UDP processing possible too, as we don't pull in unrelated code just for device discovery.
9. Our md5 convenience function should just return the full MD5. We can then implement version specific uses of it in our version specific payload processing, ie taking the substr for 3.1 signatures.
10. Essentially, TuyaFrame and the encryption utilities are constant, the only thing changing between versions is payload handling. So we can insert an extra class between these two to deal with the version specific stuff.
11. MessageParser.parse() currently checks for the frame suffix at the end of the buffer only, even if there are multiple packets. It should check for the suffix in each packet. This is a small bug but a bug nonetheless.
12. If we implement schemas, I think we should be giving developers the option to interact with the device at a low level, much as it is now. Have the "schema-tized" TuyaDevice object be a subclass of the existing TuyaDevice.
13. set() is unnecessarily complex, we should just pass the dps directly, as if the multiple option is specified. Read: set({1: true, 2: 55}) vs set({set: '1', dps: true}); set({set: '2', dps: 55})
14. Expose more device commands. Looking at MessageParsers CommandTypes there are a lot of features we can bring to developers. Even commands we don't (yet) know how to use should be accessible through some combination of TuyaDevice.send.
15. If we're breaking API anyway, we could take this opportunity to rename some things for consistency and brevity. For example commandByte could become simply command (technically not a byte anyway, it takes up a full 32 bits in the TuyaFrame even if they only use the lowest byte right now, they could easily change that in future releases).
16. Related to the above, we may want our class names to match file names. Also consider moving TuyaDevice into lib and have index.js only there to expose all the available interfaces.

[kueblc]

A lot of my suggestions amount to removing and splitting code; this should ease the testing and maintenance burden. For testing purposes I think we should gather real world samples of communication data, rather than relying on our library to convert in one direction and back in the other.

[codetheweb]

Really like all of those suggestions @kueblc, thanks for taking the time to think through this.

One other thing I was thinking about the other day was creating a second package that layers on top of TuyAPI, providing a common interface for device categories like lightbulbs and outlets. So for example, instead of encoding the color of a lightbulb whatever weird way Tuya devices expect, the user could just run await device.setColor('#ff00ff') or whatever. I don't think it should become part of the base TuyAPI package, but potentially could be very useful as a helper package.

Unfortunately, I've been pretty busy this summer and don't expect to really be able to dedicate a good chunk of time to implement some of this stuff until fall when I'm back in college. So if anyone else visits this issue and the last comment is a few months old, please have patience :).

[codetheweb]

Expanding on the above, I was thinking about this yesterday: we could move TuyAPI-related functionality into a new package, @tuyapi/driver. This package would be focused on only providing a low-level interface to devices. A second package, called @tuyapi/devices, would then provide a high-level interface (with helper functions for each device type).

Moving this package to be under the @tuyapi umbrella would have a few positives:

• Signals that I'm not the only maintainer (and maybe not even the primary one in the future)
• Allows trusted maintainers to publish updates
• Keeps the ecosystem more cohesive as all other packages are under the @tuyapi org

The main downside is that I don't see it being easy to get people to migrate to one of the new packages.

Just wanted to jot down some thoughts, take them or leave them. :)

~~Also: I've seen a few projects recently that hit API endpoints Tuya created specifically for their Home Assistant plugin. I've been meaning to try them out and see if we can get the localKey from one of them, as the AnyProxy solution doesn't seem to be working.~~ Update: doesn't seem to work after a quick inspection, the Home Assistant API only returns { online: true, state: true }.

[fondberg]

Is the anyproxy solution not working? I was just about to try tuyapi out because my latest plugs couldn't be flashed using tuya-convert

[codetheweb]

@fondberg people have had mixed results with AnyProxy from what I've heard.

You can always just sniff the key manually instead.

[codetheweb]

I've been working on a rough prototype for the next major version of TuyAPI over the last few weeks, and I think it's ready for some testing and review.

I made a new repo for it at @tuyapi/driver. I'm not completely committed to moving the repo source at this point, and this repo may get merged or something into this one once it's ready.

A few things:

• It now uses Typescript to reduce bugs, improve readability, and help with IDE auto-completions. This is my first time using Typescript, and I'm really liking it so far but I'm sure there's some basic concepts that aren't implemented/represented properly because of my inexperience. (Actually, I think @tjfontaine was the one asking about Typescript last year; I'd appreciate any feedback you have time to give.)
• ~~I'm planing to eventually use Babel for transpiling, but you'll have to use Nodejs ≥ 12.0.0 for now.~~ Edit: implemented Babel.
• Overall, my main goals for this next version are in order:
  1. Make the interface succinct and easy to use at any level - from sending raw Buffers over the socket, to generating custom Frames, to top-level get/set commands.
  2. Make the code as easy-to-understand as possible. The flow of data in the current codebase is extremely hard to understand. Making it easier to follow will help future contributors as well as those trying to port this to a different language.
  3. Better coverage of edge cases and error handling. For example, if a device returns data format unvalid we shouldn't be failing silently.

Here's an example script that logs all received packets while rapidly toggling the first property of a device (save as dev.js so it's ignored by Git, build Typescript files first with npm run build):

const {Device} = require('./dist');

const device = new Device({ip: '', key: '', id: ''});

device.connect();

device.on('connect', async () => {
  device.update();
});

device.on('data', frame => console.log(frame));

device.on('state-change', async () => {
  console.log(device.get());

  await device.set({1: !device.get()['1']});
});

I've only tested it with v3.1 devices so far, as I still haven't gotten a v3.3 device. v3.3 should work, in theory.

[kueblc]

Awesome, looking forward to checking it out, though I'm not familiar with Typescript so like you I may not be able to recommend the "right" way to do things.

[codetheweb]

Has anyone had a chance to take a look at the new package?

I'd love to ship it before the end of the year if possible.

[jezzaaa]

I don't know if this the right place to post the information but I want to let you know that I have developped a tool to dump communication with tuyadevices. It can be found here: https://github.com/py60800/tuyadump.

@py60800 : I'm fascinated by this. What would it take to add support for v3.3?

[py60800]

What are the change between protocol 3.3 and 3.1 ?

If there is any change in the way encryption is used, it may take some time (they may have fixed some errors in the implementation).

I do not have any device using this protocol but I could try to add support if I am provided with samples of communication (tcpdump).

[codetheweb]

@jezzaaa @py60800 if you want to continue this conversation please move it to an issue on @py60800's repo.

[codetheweb]

@kueblc I've been thinking about your suggestion to run end-to-end tests against real data, but I'm having a hard time thinking through how it would work. i.e. if your source is a PCAP file, when do you send the packets coming from the simulated device? How do you check if the packets match without always doing a byte-by-byte compare (sometimes packets have timestamps in the data)?

I really would like to test against captured traffic, just not sure how it would work.

[kueblc]

I think that my comment was more about cases like this where we are testing the parser against the encoder. (The idea being, if the parser and encoder are equally wrong, we wouldn't know.) Tests like these should be replaced with tests like this where the cipher is tested against a known fixed result.

We could implement this by adding PCAPs and a pcap parser/relay, but I think it would be easiest to implement by just adding more data samples embedded as strings directly to the tests.

[codetheweb]

Got it, that makes more sense.

So test against static data for unit tests and @tuyapi/stub for end-to-end?

[kueblc]

Yup, that's it. Hopefully I'll be able to help out again soon, just finishing up a couple big jobs.

[codetheweb]

Alright, sounds good.

[codetheweb]

Due to other commitments, I unfortunately don't see this getting done by the end of the year. I think January / early February is a more realistic target at this point.

The core code is mostly done, we just have tests and error handling / edge cases left (unless we want to restructure it slightly).

[Bobo-amg]

Any progress?

Thanks

[codetheweb]

TL;DR: Yes. No. Maybe?

I've been meaning to post an update for a while. I don't know if you saw it, but @tuyapi/driver is where I've been working on "v6". But I haven't worked on it in a while (4 months according to the commit log) for a number of factors:

• The current version of TuyAPI seems stable enough for the vast majority of users.
• Responding to issues sometimes feels like a part-time job on its own, it's hard to find time that I can devote to a new version.
• I actually don't use TuyAPI myself at all anymore. My original goal was to simply be able to control my devices with HomeKit through Homebridge, and that has been accomplished in other ways - like with homebridge-tuya-web, which from a user perspective is way easier to set up than messing around with linking devices to get IDs & keys.
• Lastly, in the last year or so the tuya-convert project was started. When I first started reverse engineering Tuya devices, I never imagined that it would be possible to change their firmware OTA with a relatively painless process. If I knew it was possible, I probably would have written a similar tool for replacing the firmware and never published TuyAPI in the first place. This may start going a bit into the weeds, but:
  • I've recently realized that there are really only two categories of users (feel free to correct me if I'm wrong on this). The first is those who are OK with the Tuya-provided firmware remaining on their devices and calling back home once in a while. While TuyAPI allows them to control their devices, it probably doesn't matter that much to them that it works over their local network, they would probably be fine controlling their devices by calling cloud APIs instead. They just want their devices to work with as little fuss as possible, and maybe want to retain the ability to control devices with the official app. The second category is those who want to run their own firmware on their devices, for one of two reasons: either they don't feel like they truly own their device unless they can do so, or they simply want the expanded functionality that comes with custom firmware.
  • Because of this, it's started to feel like TuyAPI is a stopgap measure - somewhere between cloud control and true device freedom; without actually offering either.

So. All that to say that I don't yet know what the future holds for TuyAPI. If I do new development work in the future, it'll probably be on cloud libraries or tuya-convert.

[gredin]

@codetheweb I certainly agree with your analysis of the users (2 categories of needs).

Which tool would you recommend for using Tuya cloud instead of local communication with the device? I think "many" people just need to use any login/password from one of the official app and still be able to interact with Tuya Cloud API.

[codetheweb]

There's already a repo for their Open API, but I haven't thoroughly documented it yet. We would also have to add a lot more functions to the wrapper to allow for device control, but that's fairly trivial.

It would be nice to be able to use your login & password from the Tuya app, but without extracting the API key from the official app you're only able to use the Home Assistant API that Tuya made available. I don't think that API can control all device types / attributes.

[TRSx80]

Came across the link here while reading through #5, and read your thoughts above:

there are really only two categories of users

I would say that I am in a third category. I would normally be in your second category, in fact I bought my devices from the outset with the intention of flashing {Tasmota,Espurna,etc.} on them, but ended up learning they were not ESP8266 based at all but rather TW-02 (WinnerMicro W600-based) smart sockets. Therefore I think the only option available to me will be to try and contain them on my own local network, and control them locally.

I am definitely NOT OK with them "phoning home" in fact if I cannot figure out a way to get this to work without doing so, I am simply going to chuck them in the bin.

I have no idea how common these "not ESP8266" based modules are, nor if they will become more common in future, but just something to factor into your consideration whether there is still a need here.

I'm not implying you need to force yourself to do anything you don't want to. You have done a lot already @codetheweb, and it is greatly appreciated! If you feel 2.0 is "good enough" then so be it. Overall your assessment of the shifting sands is more or less correct I suppose. Just wanted to point out there is a third category (since you asked). :wink:

Cheers, mate! :beers: Happy Friday! :tada:

[codetheweb]

Good point, thanks for chipping in.

If you go through Tuya's wizard to create new whitelabeled devices, there are actually a far number of hardware choices that aren't ESP* based - so I expect to continue to see non ESP* Tuya devices manufactured. I dunno how many though.

Would be very nice if there were some stats somewhere showing what chipsets active Tuya devices use as percentages.

[TRSx80]

Would be very nice if there were some stats somewhere showing what chipsets active Tuya devices use as percentages.

More data points are almost always good for decision making, I suppose. In that vein, and for the record, the following is what I came across during my research.

I remember @kueblc while back (2018?) in #5 saying ~ "API keys used to be easy to get, now they cost $1,000." Now maybe he was talking about different type of keys than current method (not sure) but that certainly does not appear to be the case right now (or I am mixing things up).

Or possibly Tuya market strategy changed. More recently they seem to want to be much more open to devs. I base this on article like this: Tuya Smart unveils 2020 strategy, launches Cloud Development Platform to global developers during the AI+IoT Business Conference which is from May 27, 2020.

Personally I am always leery of big companies marketing bullshit. I suppose maybe once I create new developer account (or perhaps independent of it) I could email them and ask what the percentages are. We will find out real fast just how "open and co-operative" they really are trying to be...