General Discussion

[jepsonrob]

I'm trying to get this to work with some smart bulbs and light strip, but currently the only missing piece for me is how you got the prefix & suffix in requests.json. Are they just the packet header/footer or is there more to it than that?

Awesome work here by the way, thanks for all your work!

[codetheweb]

Yep. To find the prefix and suffix for new devices, you'll have to fire up Wireshark and sniff the packets going to your device. The data portion of the packet starts with a version number (probably 3.1) and ends with = or == (a base 64 encoded string). The rest of the packet before and after (not including other TCP segments, like destination IP, checksum, etc.) that data string is what's plugged into the prefixes and suffixes.

I'm guessing this padding has an actual meaning, but so far I haven't been able to find anything. @blackrozes has suggested that it's just random data with a few special bytes in the beginning and end as devices seem to expect a fixed packet length. If you have any theories, I'd love to hear them.

[jepsonrob]

Yeah I've just checked in Wireshark and it fits your description perfectly.

Are you sure everything before the 00 00 55 aa part of the prefix is necessary to use with on & off? It appears be the TCP information you were talking about, and you cold be hardcoding your own private IP addresses into it (looks like your outlet is on 192.168.0.108).

As to theories, I'm noticing that the 8th hex value (starting from 00 00 55 aa) increments by 1 with every request and everything else but the very last bit is identical across the board, including in my own packets. The last bit before the data appears to only be either b3, 9b, 46 or 00 in both my own and your prefixes. Not sure what these represent at all!

[codetheweb]

Oh wow thanks. I was going to say "sorry for the confusion, I meant that you shouldn't include the TCP packet metadata", but then I looked at requests.json 🙄. Oops. I'll push an update in a few minutes removing the hex metadata.

[codetheweb]

Updated with b766439f. I've now whittled them down to this:

"outlet": {
    "status": {
      "prefix": "000055aa000000000000000a00000046",
      "command": {"gwId": "", "devId": ""},
      "suffix": "000000000000aa55"
    },
    "on": {
      "prefix": "000055aa00000000000000070000009b",
      "command": {"devId": "", "dps": {"1": true}, "uid": "", "t": ""},
      "suffix": "000000000000aa55"
    },
    "off": {
      "prefix": "000055aa0000000000000007000000b3",
      "command": {"devId": "", "dps": {"1": false}, "uid": "", "t": ""},
      "suffix": "000000000000aa55"
    }
  }

I tried tearing out the ...7... and ...a..., replacing them with 0s, but it didn't work. So they must mean something. Same with 46, 9b, b3.

Is it working with your device yet?

[fusionedv]

I think the devices are expecting a special packet size. In my script I create a buffer with this size and put the data inside like this:

var buf1 = new Buffer(171); 
buf1.write('000055aa00000000000000070000009b'+ (data),0,"hex");
buf1.write('0000aa55',167,"hex");

[jepsonrob]

Okay I've done some analysis of these packets and have figured some stuff out:

Prefix & Suffix The prefix is always (from my tests) 16 bytes long.

The first half - 8 bytes 00:00:55:aa:00:00:00:6b - of our packet is static, except for the last (8th) byte. The 8th byte increments by one every time a new command is sent, but I've seen some results that add more than this using commands that were not on/off. At any rate, it is just a counter and I think it doesn't really matter what the value is.

The second half - 00:00:00:07:00:00:00:9b - gives 2 pieces of information: the type of packet being sent and the length of the remaining the data/packet. So far I've found that the 4th byte is 07 when sending commands, 08 when receiving a reply from the device, '9e' for broadcast messages, and 0a when getting the status (haven't tested the status flag but I'm assuming this is the case from your prefix in requests.json).

The final value (the 16th byte of the whole prefix) is the size in bytes of the subsequent packet (obviously as a hex value). This is the length of the encrypted data in bytes plus the suffix - basically everything after the prefix. From my tests the suffix has always been 8 bytes.

I have no idea how the suffix is created, but it doesn't seem to change anything and I've had good results leaving it as 000000000000aa55. I'm still not quite there with getting this working on these devices yet, but knowing how the prefix work feels like most of the heavy lifting.

Connections & Broadcasts

When the device is not connected, it sends out UDP broadcast packets to the network every 3-6 seconds. This contains plaintext JSON with the following information: {"ip":"192.168.0.xx","gwId":"002003595ccfxxxxxxxx","active":2,"ability":0,"mode":0,"encrypt":true,"productKey":"AqHUMdcbxxxxxxxx","version":"3.1"}

This could be used as a way of enumerating devices on the network and all information other than the key.

Furthering the enumeration aspect, we get a plaintext list of commands available on the device from the initial connection handshake: sending the {"gwId":"002003595ccxxxxxxx","devId":"002003595ccfxxxxxxxx"} packet leads to a plaintext JSON status response from the device: {"devId":"002003595ccfxxxxxxxx","dps":{"1":true,"2":"colour","3":255,"4":255,"5":"00ff0d007bffff","6":"00ff0000000000","7":"ffff500100ff00","8":"ffff8003ff000000ff000000ff000000000000000000","9":"ffff5001ff0000","10":"ffff0505ff000000ff00ffff00ff00ff0000ff000000"}} which can in turn be used to enumerate JSON for the specific device.

One thing I've struggled with is actually decrypting the packets sent/received using the key. Has anyone got a deciphering script?

[codetheweb]

@jepsonrob thanks, that's really helpful.

Receiving Broadcasted Messages

For anyone interested, you can receive the UDP broadcasted packets with a simple script:

const dgram = require('dgram');
const server = dgram.createSocket('udp4');

server.on('error', (err) => {
  console.log(`server error:\n${err.stack}`);
  server.close();
});

server.on('message', (msg, rinfo) => {
  console.log(`server got: ${msg} from ${rinfo.address}:${rinfo.port}`);
});

server.on('listening', () => {
  const address = server.address();
  console.log(`server listening ${address.address}:${address.port}`);
});

server.bind(6666);

I may add auto discovery functionality soon, although it's not a huge priority for me so if some else wants to do it and open a pull request I'd be very grateful.

For Decrypting:

Basically, just do the reverse of this:

  // Encrypt data
  this.cipher.start({iv: ''});
  this.cipher.update(forge.util.createBuffer(JSON.stringify(thisRequest.command), 'utf8'));
  this.cipher.finish();

  // Encode binary data to Base64
  const data = forge.util.encode64(this.cipher.output.data);

  // Create MD5 signature
  const preMd5String = 'data=' + data + '||lpv=' + this.version + '||' + this.key;
  const md5hash = forge.md.md5.create().update(preMd5String).digest().toHex();
  const md5 = md5hash.toString().toLowerCase().substr(8, 16);

  // Create byte buffer from hex data
  const thisData = Buffer.from(this.version + md5 + data);
  const buffer = Buffer.from(thisRequest.prefix + thisData.toString('hex') + thisRequest.suffix, 'hex');

I'll try to add a specific decryption function within the next couple days.

[clach04]

@jepsonrob when you say, " not connected, it sends out UDP broadcast". Do you mean on the network but no devices are connected to it?

I tried both @codetheweb 's suggested js listener script and a Python script based on SSDP discovery code on port 6666 and I'm not seeing anything.

BTW thanks for excellent write ups, I now have a Python version, its rough and ready but works :-) https://github.com/clach04/python-tuya

One thing I've noticed is that if there are spaces in the json payload, the device will not respond. Not an issue under js, but the Python stdlib library adds spaces.

[jepsonrob]

@clach04 - that's exactly what I mean, yep. It's potentially useful for enumeration because all devices on the network can see these packets in plaintext and they contain a bunch of useful information.

And your issue with the JSON payload & spaces might come from the last 2 characters in the prefix - it needs to be the total length of the rest of the payload in hexadecimal. Using spaces will increase the size of the payload and if the value isn't the correct size then the command goes ignored.

[clach04]

@jepsonrob thanks for the length info, that's it for sure. I added support for this in https://github.com/clach04/python-tuya/commit/fc4612f168b5a1f0f6ca2396dbdea7e2cc10fe7b - I'm not sure if its a single byte for length or not (I've only coded it for a single byte and I've not yet added sanity check for that in case the payload goes above 255 bytes).

Thanks also for confirming about broadcast. Sadly, as per my previous comment, I'm not seeing this on my network (using code above i.e. something sitting on port 6666, I've not tried wiresharking the network). Any pointers?

I've created a wiki https://github.com/clach04/python-tuya/wiki to store progress and other useful related docs. E.g. I figured out the timer for the SM-PW701U device.

[jepsonrob]

@clach04 - try using this python script and see if anything comes through: it's working for me!

from socket import *
s=socket(AF_INET, SOCK_DGRAM)
s.bind(('',6666))
m=s.recvfrom(1024)
print m[0] 

And yeah I've only had it working for a single byte which is driving me crazy because the payload I need is over 255. Interestingly though, when I'm looking at the packets sent in Wireshark there's never anything over 255 bytes in size! I feel like this means I'm getting something wrong with my JSON payload and it's coming in too large, but I can't quite figure out how to decrypt the outgoing packets so I can't see the intended behaviour.

Nice work on the python version by the way! That wiki is a useful resource - I'm going to write a no-stupid-questions type of tutorial for this we've got it working with other devices, probably just to help people with IFTTT integration, and I'll link out to that for further reading.

[clach04]

@jepsonrob that script worked fine. My script also worked fine this morning (not sure why, maybe too tired last night).

Interesting notes:

• I'm testing on Windows at the moment
• python script only works if I'm on the WiFi network, being on the same network via a wire doesn't work (I thought I tested this last night but maybe was not testing with py2) - EDIT looks like this was a Windows firewall issue, specific Python process blocked on certain networks
• python script only works for me with Python 2 (change print to add parens for py3), under Python 3 I do not see any packets - this one still confuses me. Not sure if this my machine, Windows, etc. - EDIT looks like this was a Windows firewall issue, specific Python process blocked on certain networks
• node script above does NOT work for me (possible same root cause as py3 failure?) - EDIT gonna guess this was a firewall issue too

@jepsonrob script (slightly modified):

from socket import socket, AF_INET, SOCK_DGRAM

s = socket(AF_INET, SOCK_DGRAM)
s.bind(('', 6666))
m = s.recvfrom(1024)
print(repr(m[0]))

My script:

# discover devices
# NOTE for my devices this only works with Python 2.6
# py3.6.1 runs but never reports packets

import socket
import struct


host_port = 6666
host_ip = '239.255.255.250'

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, 2)
sock.bind(('', host_port))

mreq = struct.pack('4sl', socket.inet_aton(host_ip), socket.INADDR_ANY)
sock.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)

try:
    while True:
        print('listening')
        data = sock.recvfrom(1024)
        raw_bytes, peer_info = data
        print(data)
finally:
    sock.close()

[clach04]

@jepsonrob for decrypting packets, try this (needs my Python module, https://github.com/clach04/python-tuya):

import base64
import time

import pytuya


key = 'YOUR_KEY_HERE'
packet = 'Python raw bytes'


PREFIX_LEN = 16
SUFFIX_LEN = 8
prefix = packet[:PREFIX_LEN]
json_crypted_plus_suffix = packet[PREFIX_LEN:]
json_crypted_len = ord(prefix[-1])
json_crypted_b64_with_header = json_crypted_plus_suffix[:json_crypted_len-SUFFIX_LEN]
version_str = json_crypted_b64_with_header[:3]
print('version_str (%d) %r' % (len(version_str), version_str))
hexdigest = json_crypted_b64_with_header[3:][:16]
print('hexdigest (%d) %r' % (len(hexdigest), hexdigest))
json_crypted_b64 = json_crypted_b64_with_header[3+16:]
suffix = packet[-SUFFIX_LEN:]  # assume we know length of end


print('packet (%d) %r' % (len(packet), packet))
print('prefix (%d) %r' % (len(prefix), prefix))
print('json_crypted_plus_suffix (%d) %r' % (len(json_crypted_plus_suffix), json_crypted_plus_suffix))
print('json_crypted_len %d' % (json_crypted_len))
print('suffix (%d) %r' % (len(suffix), suffix))

print('json_crypted_b64 (%d) %r' % (len(json_crypted_b64), json_crypted_b64))
cipher = pytuya.AESCipher(key)
json_raw = cipher.decrypt(json_crypted_b64)
print('json_raw (%d) %r' % (len(json_raw), json_raw))

EDIT only supports 255 lengths (could be updated if it turns out protocol supports it.

[codetheweb]

FYI, the UID is now no longer required when constructing an instance thanks to @jepsonrob.

[Marcus-L]

I read through #2 where accessing the Tuya API directly was briefly discussed. It would be preferable to be able to access the device IDs and localKeys without having to sniff traffic (especially since this appears to no longer be possible in recent versions of Android with the Packet Capture app).

It is possible to retrieve the Android App's API Key and Secret from the app source code. I believe these are the values:

However, I attempted to use the API documentation but was unable to successfully authenticate. Here's a fiddle that I was working with: https://dotnetfiddle.net/QzIVrP

I think the problem may be with the signing. When I logcat the Tuya Android app I see that the "sign" parameter is 40 characters long, however according to the documentation it should only be 32 characters since it's just a hex string representing a 128-bit MD5 hash.

Anyway, not sure if you'd want to fiddle around with it any more, but it really would be nice for the API to be able to fetch the device localKeys.

[clach04]

@Marcus-L keep us posted. I'm going to be a bit of a nay-sayer and say that I suspect we do not need API access to the cloud to register the device and get an encryption key. My gut tells me that registration of the device can be figured out.

RE packet capture problems, can you clarify which app/version and the problems you had?

I'm using https://play.google.com/store/apps/details?id=com.xenon.jinvoo&hl=en v1.0.3 which appears to be the latest and still works for me with https://play.google.com/store/apps/details?id=app.greyshirts.sslcapture&hl=en - but I'm thinking I should back it up and post here!

[clach04]

Forgot to add there is a chance the key is usable client side to interact with devices?

I've not tried to decompile it (and probably won't have time), is that how you made this progress?

[codetheweb]

<rant> Hey guys, just before you get any farther I wanted to make a quick comment. I will not allow working API keys extracted from the source code of other companies to be hard coded into my project. If you can use them to discover more information about the API and controlling the device locally, all well and good. But please don't expect me to add them. It's about ethics, not whether it's possible.

Besides, this was intended to be a project to control devices entirely locally - no calling back to home. </rant>

Sorry if that came off as a bit too stern. Maybe you're not planning to do that :). In any case, good luck mapping out the API.

[Marcus-L]

No worries @codetheweb, wasn't planning on hard-coding those keys into anything, Just exploring the API possibilities. As @clach04 mentioned it's likely to be possible to do everything locally. None of this would be necessary of course if the manufacturers would just release open tools, some documentation or even just rough specs! Or if the Tuya or Jinvoo app were to show the localKey settings. But here we are.

[codetheweb]

OK, thanks for the clarification. 😀

On Dec 6, 2017, at 11:50, Marcus Lum [email protected] wrote:

No worries @codetheweb, wasn't planning on hard-coding those keys into anything, Just exploring the API possibilities. As @clach04 mentioned it's likely to be possible to do everything locally. None of this would be necessary of course if the manufacturers would just release open tools, some documentation or even just rough specs! Or if the Tuya or Jinvoo app were to show the localKey settings. But here we are.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[Marcus-L]

No problem, thanks for the useful protocol info. I did a rough port of the api to .NET Standard here: https://github.com/Marcus-L/m4rcus.TuyaCore

I poked around a bit to try to see if I could find any more info on the discovery/localKeying process but didn't come up with anything useful.

[nijave]

If you have a rooted Android phone, you can retrieve the settings from the app (Smart Life) data storage. The keys/configured devices are located at /data/data/com.tuya.smartlife/shared_prefs/dev_data_storage.xml

There's a string in there (the only data) called "tuya_data". You need to html entity decode the string and it contains a JSON string (yes, this is slightly ridiculous). Inside the JSON string are the keys.

[abarvinsky]

Hi everyone, I just bought a couple of the outlets that work on this protocol but when I try to sniff the traffic using Charles, I don't get any GET requests like the ones mentioned here but only a few POST requests but it looks like the traffic is encrypted. I added tuyaus.com to the SSL Proxy in Charles but it did not help. I am not sure if it is the SSL encryption or Tuya is using their own encryption but it definitely looks like they changed the API or something like that.

Or maybe I am doing something wrong. Any input is appreciated

[clach04]

@nijave I'm not sure where you read GET. I can definitely confirm you are looking for POST (I'd recommend looking at the response to a POST). See https://github.com/codetheweb/tuyapi/blob/master/docs/SETUP.md - someone did comment stating they thought an app didn't contain the information any more. What app and device are you using? See my post above with the word "xenon" for where I know it still works.

[Exilit]

The documentation from tuya themselves states the following:

Encryption key: The related interface key before activation is the first 16 characters of the authKey of the device. The key of the relevant interface after activation is secKey.

Moreover it says:

...	...
secKey	Device and cloud communication key, obtained through the cloud-activated devId
localKey	The device communicates data with the cloud MQTT server AES encryption key, which is returned with the secKey when the devId is activated in the cloud

If I get it right this means that there is a preset device specific key (the authKey). For the first communication during activation the first 8 byte(??) are used as AES encryption key. During the activation two other keys - the localKey AND the secKey- should be transmitted to the device. From there on secKey is used for HTTPS communication whereas localKey is used for MQTT communication.

But that can't be correct as I cannot see the secKey anywhere. Moreover if it is correct, how do you think does the App know the authKey that is used for activation?

If the original app is able to retrieve the authKey from the devices, we should be able to do so also, don't we?

[codetheweb]

Perhaps, but I doubt we can without also having an API key.

[nijave]

There appears to be some documentation here on how the protocol works that communicates with these https://docs.tuya.com/en/mcu/mcu-protocol.html

Sniffing traffic with my phone it looks like sending a "09" in the prefix to get the status will set up a sort of stream where the outlet keeps sending status updates every couple seconds

[clach04]

https://docs.tuya.com/en/mcu/mcu-protocol.html - looks like its for serial access (I don't read Chinese, I'm relying on google translate). The pictures and screen shots on the "MCU Debug" pages https://docs.tuya.com/en/mcu/FAQ/log.html and https://docs.tuya.com/en/mcu/debug_assistant.html seem to back that up. Its possible some of the same protocol is used but there is no encryption mentioned (presumably as serial means physical access).

[clach04]

Schemas - I've not had chance to try the schema code out in this project but I have taken a look at the schema information that gets stored on the android device. There is some really useful stuff there, e.g. the timer (which it turns out is called countdown).

schema = [
    {
        "code": "switch_on", 
        "name": "\u5f00\u5173", 
        "iconname": "icon-dp_power2", 
        "mode": "rw", 
        "property": {
            "type": "bool"
        }, 
        "type": "obj", 
        "id": 1, 
        "desc": ""
    }, 
    {
        "code": "countdown", 
        "name": "\u5012\u8ba1\u65f6", 
        "iconname": "icon-dp_time", 
        "passive": True, 
        "mode": "rw", 
        "property": {
            "scale": 0, 
            "min": 0, 
            "max": 86400, 
            "step": 1, 
            "type": "value", 
            "unit": "\u79d2"
        }, 
        "type": "obj", 
        "id": 2, 
        "desc": ""
    }
]

There are also URLs for icons.

See https://github.com/codetheweb/tuyapi/pull/16 for some details (and a script).

[nijave]

Right. I was thinking that maybe the TCP protocol is a wrapper around a serial connection. It seems they use the same delimiters "55aa" and all the 0s that follow are items not set. One of them mentioned a variable length data section in the request. That would also explain why it's so picky about things like spaces in the JSON if it's just doing a naive marker search and not really processing JSON.