emporia-vue2-reversing
emporia-vue2-reversing copied to clipboard
flaviut
Decoding sensor data
Repurposing this issue for the discussion that it started. Was originally a question about connecting, and turned into a discussion about deciding the sensor data.
In the blog post, you mentioned that one of your USB-Serial adaptors wasn't working when you tried to connect to the ESP32. I'm wondering if I'm hitting the same issue; I only have one serial adaptor to test with at the moment though.
I'm using the FT232RL FTDI Mini USB to TTL Serial Converter, and getting nothing while trying to connect:
> esptool.py --port /dev/cu.usbserial-A50285BI read_flash 0 0x400000 flash_contents.bin
esptool.py v2.8
Serial port /dev/cu.usbserial-A50285BI
Connecting........_____....._____....._____....._____....._____....._____....._____
A fatal error occurred: Failed to connect to Espressif device: Timed out waiting for packet header
Process I'm taking to enter flashing mode:
- Power with 3.3v
- Pull IO0 pin to GND
- Pull EN pin to GND, and release
- Release IO0 pin
But that's not working; my only thought is that maybe there is something else connected with TX/RX that is interfering? And a different serial converter could sort through the noise?
Any guidance would be appreciated!
Edit by @flaviut for visibility:
There is now a ESPHome component. You can follow development at
- https://github.com/krconv/esphome/tree/add-emporia-vue-maelstrom96
- https://github.com/Maelstrom96/esphome/tree/add-emporia-vue2-staging
- https://github.com/esphome/esphome/pull/2871
- https://github.com/esphome/esphome-docs/pull/1702
Actually, I had TX and RX backwards; TX on the debug header goes to RX on the ESP32. My bad (you mentioned it in the blog post and I missed it). Just successfully downloaded the flash and got debug logs
Ha! This stuff happens, glad you got it figured out! Please feel free to ask anything else, and let me know if you end up building something on top of all this!
My main goal is to build an esphome integration. I've never had to fully reverse a chip before, but it feel like a fun challenge that I might be able to help with, and I like the small form-factor and low price of the Vue. I'd appreciate any pointers if you have any; currently waiting for a USB oscilloscope to help with identifying what each of the pins are doing.
Sure! First off, oscilloscopes are very helpful for identifying the function of things, but when it comes to actually reading digital data you want a logic analyzer.
Anyway, please do poke at it, it's a lot of fun. If you'd like though, I briefly described the physical communication on the board here: https://flaviutamas.com/2021/reversing-emporia-vue-2#on-board-communications
There's another chip on the board that communicates with the ESP32 over I2C. The raw I2C message gets dumped over MQTT as hex here: https://github.com/flaviut/emporia-vue2-reversing/blob/master/example_messages.txt#L2-L19. Note that right below that hex data are the output results: ~the I2C data is combined with the calibration data stored in NVS to produce the final output:~ The calibration data in NVS is nulled and not used at all.
By the way, I've given you write access to this repo! Please feel free to do whatever you wish with that access, but let's keep Emporia-copyrighted stuff off of here to avoid causing them problems :)
Hey - I'm actually working on the exact same goal as @krconv. I'm mostly trying to make it so I can add a separate ESP32 running ESPHome that would read the messages on the i2c bus and publish that data to MQTT or through the native ESPHome API, without the need to flashing the onboard module. The reason being that I like having the data in their UI, but I also want to be able to do second triggers on the energy consumption. At first I was thinking about making a MQTT bridge to their AWS IOT mqtt endpoint, but I wasn't able to reverse engineer the certificates/auth method.
Because of this, I've been working on reverse engineering the i2c message data and I think I mostly have it figured out at this point. Here is my data spreadsheet I used to figure out / validate the parsing logic: https://docs.google.com/spreadsheets/d/1Yu98tDRhFJqqMntNak6X297SKyzQtf0_Cvv_g7qKewk/edit?usp=sharing.
The only data missing from the message is the calibration - which is only on the esp32 - and the voltage frequency.
Edit: Forgot to say, thanks @flaviut for the post and the data in the repo, it was immensely useful. Also, I updated my Vue2 firmware to the last version (Vue2-1612298550, Tue Feb 2 20:42:33 UTC 2021) and it's still outputting logs and the TEST_MODE is still there and it looks like it's still working.
WOW! This is incredible! I'm curious about how you went about doing this, I did spend a while looking at things in a hex editor, but didn't really get anywhere.
It took me a bit to understand the spreadsheet, but now that I do, I've written up a C declaration for that data:
// all data is little-endian
struct __attribute__((__packed__)) ch_power {
int32_t ph1, ph2, ph3;
}
struct __attribute__((__packed__)) vue2_i2c_data {
uint32_t unknown1;
ch_power power[19];
int16_t ph_voltage[3];
int16_t frequency;
int16_t ph2_offset_deg;
int16_t ph3_offset_deg;
int16_t current[19];
uint16_t unknown2;
}
One thing that I suspect is going on here (from my use of the Vue 2 locally) is a non-1:1 mapping of channel numbers to data indexes. I think, but am not sure, that some of the indexes here do not match up with the numbers printed on the box.
I'll start off by saying that I'm mostly out of my league when it comes down to reverse engineering things, but I wanted to give it a shot anyway.
Input P[V] values
The first thing I did was to manually check the debug/v2 message that you had in your post to see if I could find any pattern. An obvious one was the amount of FFFF value bytes separated by 2 bytes of data. I also noticed that all the P[V] values were small-ish negative ones. Doing a find and count for FFFF in Notepad++ returned 57 match. Dividing that value with the 3 P[V] per input results in 19 - the same amount of input we have in the logs. That block was then clearly the data for all the input power values.
I copied the first 12 bytes (which should correspond to the first 3 P[V] of I01) in this amazing hex tool and started to check for other patterns in the data. I also copied another 12 bytes from another message with positive values, and the values seemed to correlate when parsed as INT32 (DCBA).
I wasn't sure what to do then, since the decimal values weren't the same as the expected values. I decided to plot values from multiple inputs on a graph to see if there was a linear equation hidden somewhere.
Here's what it gave me :
As you can see, it does look like it's linear, but calculating the actual expression gave me a pretty significant offset (which can also be seen on the graph since the line is not going through 0). A small offset is expected since the values are most likely rounded, but the result is too much. Also, by looking at the graph, we can see that there are 2 general groups. I was able to see that the slope value was different between I01-I03 and I04+.
After thinking about it, and checking the points (expected value and RAW decimal) on the graph, I noticed that not all the points were exactly on the trend line, so I started to wonder if the calibration numbers could be at play, since it would explain the deviation. I decided to multiply the values by their respective calibration and plot those values.
Here's the graph after that :
As you can see, the deviation became negligible, and the offset was almost 0 (still some but very small, > 0.03). This told me I should now be able to calculate the slope and then use that to get the P[V] value. The slope 5.5% for the first 3 inputs, and 22% for the rest.
Voltage ADC
After that, I was wondering what other values I could find, and decided to enter the next couple of bytes into the HEX tool, and got three values that I could spot easily in the debug output:
| pos | Raw | INT16 |
|---|---|---|
| 0 | 00 0F | 15 |
| 2 | 00 0C | 12 |
| 4 | 00 0B | 11 |
V1: 0.3, 0.4 Hz, *15*, 0.0225336
V2: 0.3, 0 degrees, *12*, 0.0220000
V3: 0.2, 0 degrees, *11*, 0.0220000
I wasn't sure what those values where, but I decided to see if multiplying this with the calibration data would actually give the voltage, and it actually did. So I figure this is a voltage ADC value.
Input Current?
At this point, the amount of bytes of data was pretty low, not enough for 19 x int32, I also noticed that the voltage ADC was an int16, so I started to see if I could notice any pattern with 19 x int16 in a row. It was pretty easy to spot since all the values were pretty much the same, see for yourself :
0050 0050 0050 FD4F FD4F FD4F FD4F FD4F 0350 0350 0350 FD4F FD4F FD4F FD4F FD4F 0350 0350 0350
I01: 372.4, P[V1]: -80.0, P[V2]: -21.9, P[V3]: -4.9
I02: 372.4, P[V1]: -80.0, P[V2]: -21.9, P[V3]: -4.9
I03: 372.4, P[V1]: -80.0, P[V2]: -21.9, P[V3]: -4.9
I04: 93.1, P[V1]: -19.4, P[V2]: -5.2, P[V3]: -0.9
I05: 93.1, P[V1]: -20.3, P[V2]: -5.6, P[V3]: -1.7
I06: 93.1, P[V1]: -20.5, P[V2]: -5.8, P[V3]: -1.3
I07: 93.1, P[V1]: -19.2, P[V2]: -4.4, P[V3]: -2.4
I08: 93.1, P[V1]: -20.0, P[V2]: -5.7, P[V3]: -0.7
I09: 93.1, P[V1]: -20.9, P[V2]: -4.8, P[V3]: -1.0
I10: 93.1, P[V1]: -19.4, P[V2]: -6.2, P[V3]: -1.0
I11: 93.1, P[V1]: -20.1, P[V2]: -6.1, P[V3]: -0.8
I12: 93.1, P[V1]: -19.4, P[V2]: -5.2, P[V3]: -0.9
I13: 93.1, P[V1]: -20.3, P[V2]: -5.6, P[V3]: -1.7
I14: 93.1, P[V1]: -20.5, P[V2]: -5.8, P[V3]: -1.3
I15: 93.1, P[V1]: -19.2, P[V2]: -4.4, P[V3]: -2.4
I16: 93.1, P[V1]: -20.0, P[V2]: -5.7, P[V3]: -0.7
I17: 93.1, P[V1]: -20.9, P[V2]: -4.8, P[V3]: -1.0
I18: 93.1, P[V1]: -19.4, P[V2]: -6.2, P[V3]: -1.0
I19: 93.1, P[V1]: -20.1, P[V2]: -6.1, P[V3]: -0.8
I wasn't sure why there were 2 different values FD4F & 0350 for what looked like 93.1, but I was still pretty sure that they were the right bytes. I tried to multiply one of the values with it's input offset annnnnnnnnnnd:
0x0050 = 20480
20480 x 5.5 = 112,640
Well, that doesn't look anything close to 372.4... How about a division?
20480 / 5.5 = 3,723.63636363636[...]
That was mostly pure luck to find that, but it looks like it is indeed how it's calculated. Just need to shift the period and we're golden!
Voltage degrees
I know it's pretty useless and not needed, but at this point I feel like superman, so I wanted to see If I could crack the code.
The amount of bytes left is pretty limited : 03EC 52EA [...] A601 8E00 0000 [...] 0000
There's only a single block with 3 int16, so it's only logical that A601 8E00 0000 are the right ones, but converting them didn't really give useful numbers :
V1: 120.2, 61.6 Hz, 5241 ADC, 0.0229308 Calibration
V2: 121.3, 121 degrees, 5574 ADC, 0.0217630 Calibration
V3: 8.1, 0 degrees, 369 ADC, 0.0220000 Calibration
| pos | Raw | INT16 |
|---|---|---|
| 0 | 01 A6 | 422 |
| 2 | 00 8E | 142 |
| 4 | 00 00 | 0 |
The 0 degree made sense, but how about the 142? How does that become 121 degrees? I wondered if the values were a ratio of 360°. Calculating it actually gives the right value.
142 x
--- = ----
422 360
x = 360 x 142 / 422
x = 121.13[...]
I haven't figured out if the voltage frequency is in the data, but I don't think it is, since the first 4 bytes seems to be some kind of counter, and the last 2 bytes are always 0x0000. It could actually be the 422 value, but I would need to have more data with variations to confirm it.
One thing that I suspect is going on here (from my use of the Vue 2 locally) is a non-1:1 mapping of channel numbers to data indexes. I think, but am not sure, that some of the indexes here do not match up with the numbers printed on the box.
If you have a message that you think doesn't match the mapping, let me know. I don't believe that it will deviate from this structure but we never know.
@flaviut I wanted to know - do you happen to know the i2c frequency? I'm trying to connect to it but I'm not having much luck at 50kHz or 100kHz.
@Maelstrom96 I believe it's 100kHz. I uploaded a dump from sigrok pulseview to https://github.com/flaviut/emporia-vue2-reversing/blob/master/i2c%20dump.bin, and unless the time base is off in that, it shows 100kHz.
That hopefully the screenshot will also explain how it expects to be asked for data.
This is awesome! I just had fun decoding the message too, and it's cool to see that I landed in the same place. I used Python to help me walk through the message, activating each of the inputs and finding the bytes that changed. In case your curious, this is my helper script. I was just about do try to figure out the calibration as well, and ended up with a linear relationship (just eye-balled it, didn't verify it was exactly linear) between points on I04[V2], similar to you.
From my testing, I'm pretty sure the uint16 with offset 0x00EE is the frequency; it was sandwiched between the AC voltages and degrees, and it was consistently changing with the frequency output in MQTT. The conversion for my device is 0.0014268004584473604 * <sensor_value> + 0.4 = frequency; tried it on the example messages @flaviut and it was a little off, but I think that is due to a different calibration.
For calibration data; I think we could do the same thing that the existing CT clamp sensor in ESPhome does, where whoever is flashing the device would need to manually calibrate each of the sensors. Maybe we could reverse engineer the calibration data from nvs, and make that easier? I think it's worth skipping that for now.
@Maelstrom96 Would you want to work together on the integration? I'm not familiar with building an ESPHome integration, are you? I am familiar with C++ and think I could make faster progress on the integration though if you want to keep looking into the connections on the board because I'm not as familiar with the signal processing side of things.
And I'm pretty excited...I feel like this will be a huge improvement for cost effective energy monitoring if we can get it to work
@krconv I'm not really familiar with ESPHome components. I tried to see how ESPHome handles I2C devices and I'm not sure if we're going to have an issue with the TwoWire lib that they're using. The reason being that is looks like requests read buffer is limited to 32 bytes with TwoWire, and we need to query the full 284 bytes all at once.
I've been trying to learn how various ESPHome components work. I think under the hoods, i2c will use the TwoWire library by default, but we can configure it to talk to the onboard i2c controller instead and I'm guessing that's how the the original software does it.
I'm not sure what other implications this has, but this is an example from here to tell ESPHome to use the native ESP API:
esp32:
board: esp32-c3-devkitm-1
framework:
type: esp-idf
version: recommended
We'll most likely have an issue with https://github.com/espressif/arduino-esp32/blob/master/libraries/Wire/src/Wire.h too. And if you look at Wire.cpp, the requestFrom() implementation cast down the "len" value to an uint_8, which doesn't go high enough for our 284 bytes.
I tried out that approach, and agreed I don't think it's much of an option. I think setting the framework type to esp-idf will cause it to load this driver (instead of any of the Arduino ones), and from what I can tell it doesn't have any internal buffers, which I think is good. But, setting the framework type also causes everything to attempt to compile using the native framework, and there isn't one for MQTT according to esphome, so validation throws this error:
From what I understand, Arduino is a generic framework, and arduino-esp32 are the bindings written to port Arduino to ESP32. ESPHome just barely added support for using the native esp-idf framework in release 2021.10. It seems like if we could get @flaviut's PR merged to the esp8266 and esp32 Arduino libraries, we could keep doing I2C through Arduino, or if we have enough support on esp-idf, we could go in that direction (because this component would only need to be used on esp32 anyways).
Also, I've been thinking about a potential Config format; have either of you tried that out yet? Any thoughts on this idea?
Config Idea
sensor:
- platform: emporia_vue
update_interval: 2s
phases:
- id: phase_a
phase_input: BLACK # i.e. the color of the wire that is connected to the phase
- id: phase_b
phase_input: RED
power:
- id: total_phase_a
phase_id: phase_a # not sure how the factory software figures out which phase to use; this makes it explicit
ct_input: A # uses A-C, 1-16 as labeled on the outside of the Vue
filters: # and each one would have a separate calibration
- calibrate_linear:
- 0 -> 0
- 2393 -> 2.0
- id: kitchen_outlets_power
phase_id: phase_a
ct_input: "1"
filters:
- calibrate_linear:
- 0 -> 0
- 2393 -> 2.0
- id: stove_power
name: Stove Power
phase_id: phase_b
ct_input: "2"
filters:
- calibrate_linear:
- 0 -> 0
- 3438 -> 2.0
- lambda: return x * 2;
I've been testing out config validation on my fork using this external component:
external_components:
- source: github://krconv/esphome@add-emporia-vue
components: [ emporia_vue ]
refresh: 1min
I haven't tested anything on hardware yet, but my next step might be to connect a spare ESP32's i2c port to the Vue and see if it can talk to the sensor at all with the esp-idf library
cal_data is a false lead--the calibration data is internal to the ESP32 for their wifi hardware.
The real calibration constants are cReal and cApparent.
However, the values here are surprising. The interesting data is marked as erased, while the uninteresting version is not:
{
"entry_state": "Erased",
"entry_key": "cReal",
"entry_data": "AAAgKJkOI0EAAOC+axEjQQAA8OEE2SFBAACAbhjnAkEAAHCC3dECQQAAhAty2wJBAADg+pYAA0EAAKhAYtoCQQAA2O8s5wJBAAAkLELwAkEAADA5JMoBQQAAtLRw6wJBAAC4VIYKA0EAACjF59kCQQAAYHSiBwNBAAAwksz1AkEAALj8Wt8CQQAA6OxK6AJBAAAQYf0SA0E="
},
{
"entry_state": "Erased",
"entry_key": "cApparent",
"entry_data": "AACA/dMMwEAAAACbBxDAQAAAgB+DMcFAAAAAb/v/n0AAAACo+dufQAAAABPd7p9AAACAwU0VoEAAAACnb+ifQAAAAAaV/Z9AAACAftMIoEAAAIAUqTShQAAAAJyuA6BAAAAAsfAboEAAAACm7+ifQAAAgOacGaBAAAAAIswKoEAAAAB68PKfQAAAAEllAaBAAACAiX0koEA="
},
{
"entry_state": "Erased",
"entry_key": "cReal",
"entry_data": "AAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEA="
},
{
"entry_state": "Written",
"entry_key": "cApparent",
"entry_data": "AAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEA="
},
Decoding these gets you basically what you expect: 19 entries, the first 3 of which are different than the rest.
$ base64 -d | xxd
AAAgKJkOI0EAAOC+axEjQQAA8OEE2SFBAACAbhjnAkEAAHCC3dECQQAAhAty2wJBAADg+pYAA0EAAKhAYtoCQQAA2O8s5wJBAAAkLELwAkEAADA5JMoBQQAAtLRw6wJBAAC4VIYKA0EAACjF59kCQQAAYHSiBwNBAAAwksz1AkEAALj8Wt8CQQAA6OxK6AJBAAAQYf0SA0E=
00000000: 0000 2028 990e 2341 0000 e0be 6b11 2341 .. (..#A....k.#A
00000010: 0000 f0e1 04d9 2141 0000 806e 18e7 0241 ......!A...n...A
00000020: 0000 7082 ddd1 0241 0000 840b 72db 0241 ..p....A....r..A
00000030: 0000 e0fa 9600 0341 0000 a840 62da 0241 [email protected]
00000040: 0000 d8ef 2ce7 0241 0000 242c 42f0 0241 ....,..A..$,B..A
00000050: 0000 3039 24ca 0141 0000 b4b4 70eb 0241 ..09$..A....p..A
00000060: 0000 b854 860a 0341 0000 28c5 e7d9 0241 ...T...A..(....A
00000070: 0000 6074 a207 0341 0000 3092 ccf5 0241 ..`t...A..0....A
00000080: 0000 b8fc 5adf 0241 0000 e8ec 4ae8 0241 ....Z..A....J..A
00000090: 0000 1061 fd12 0341 ...a...A
$ base64 -d | xxd
AAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEA=
00000000: 0000 0000 0000 2440 0000 0000 0000 2440 ......$@......$@
00000010: 0000 0000 0000 2440 0000 0000 0000 2440 ......$@......$@
00000020: 0000 0000 0000 2440 0000 0000 0000 2440 ......$@......$@
00000030: 0000 0000 0000 2440 0000 0000 0000 2440 ......$@......$@
00000040: 0000 0000 0000 2440 0000 0000 0000 2440 ......$@......$@
00000050: 0000 0000 0000 2440 0000 0000 0000 2440 ......$@......$@
00000060: 0000 0000 0000 2440 0000 0000 0000 2440 ......$@......$@
00000070: 0000 0000 0000 2440 0000 0000 0000 2440 ......$@......$@
00000080: 0000 0000 0000 2440 0000 0000 0000 2440 ......$@......$@
00000090: 0000 0000 0000 2440 ......$@
I bet that these can be decoded as:
struct __attribute__((__packed__)) calib_entry {
int32_t offset, slope;
}
typedef calib_entry[19] calibration;
@Maelstrom96 You're right--increasing the number there won't work unless we adjust the function signatures too. We already have https://github.com/espressif/arduino-esp32/blob/master/libraries/Wire/src/Wire.h#L98, and it's a real shame that that function is written in terms of the other functions rather than having all the other functions call it. However, it seems like a simple-enough refactoring, and the ESP32-arduino project seems responsive to contributions.
@krconv
framework type to esp-idf will cause it to load this driver (instead of any of the Arduino ones), and from what I can tell it doesn't have any internal buffers, which I think is good. But, setting the framework type also causes everything to attempt to compile using the native framework
:(
I guess you could try adding MQTT support to the esp-idf version of the esphome project, that'd probably be something useful for everyone. But I think the fastest & easiest way would be to do PRs to the wire library, since they're pretty simple. I can do that over the next day or two!
Also, I've been thinking about a potential Config format; have either of you tried that out yet? Any thoughts on this idea?
Huh. Neat! I had no idea ESPHome made things that easy, that's really nice!
The calibration data I think will be helpful, the more I think about it. Calibrating 19 CT clamps manually might be a real pain; maybe we could provide a script that helps parse the calibration data later on.
That sounds great; I agree getting MQTT on esp-idf would probably be helpful for everyone, but ya it might slow things down. I also checked the api component, and that relies on Arduino too; so there would be no way to get the sensor data to home assistant without some more development. I did find a native MQTT library; if we do go that route, we might be the first integration that requires esp-idf, which might have some other implications. I'll take a look at the work involved to support MQTT on esp-idf and see if the i2c using the native library even works for this scenario.
So I did some digging, and here's how things work:
The NVS calibration on the Emporia Vue 2 that I got isn't used. I'm not sure why the cReal and cApparent NVS values exist, but they are set to zero and not used. The last bit of each entry is probably a valid bit, because it is 0 in all of the zeroed entries, and 1 in the original, non-zero entries.
However, that doesn't mean the Vue 2 has no calibration at all. After all, we saw that it does have some sort of calibration constants baked into the firmware from @Maelstrom96's message reverse engineering efforts. What it does have is the following array:
FLOAT_ARRAY_ARRAY_3f40e8f0 XREF[3]: 400d0f50(*),
calibration:401096c3(*),
calibration:401096c9(*)
3f40e8f0 ac 35 b9 float[8]
3c c6 6d
b4 3c 58
3f40e8f0 ac 35 b9 3c c6 float[3] [0] XREF[3]: 400d0f50(*),
6d b4 3c 58 39 calibration:401096c3(*),
b4 3c calibration:401096c9(*)
3f40e8f0 ac 35 b9 3c float 0.0226086 [0] XREF[3]: 400d0f50(*),
calibration:401096c3(*),
calibration:401096c9(*)
3f40e8f4 c6 6d b4 3c float 0.022025 [1]
3f40e8f8 58 39 b4 3c float 0.022 [2]
3f40e8fc ac 35 b9 3c 28 float[3] [1]
be b5 3c c9 5f
b5 3c
3f40e8fc ac 35 b9 3c float 0.0226086 [0]
3f40e900 28 be b5 3c float 0.0221854 [1]
3f40e904 c9 5f b5 3c float 0.0221404 [2]
3f40e908 73 53 b9 3c 58 float[3] [2]
39 b4 3c da c5
b4 3c
3f40e908 73 53 b9 3c float 0.0226228 [0]
3f40e90c 58 39 b4 3c float 0.022 [1]
3f40e910 da c5 b4 3c float 0.022067 [2]
3f40e914 62 98 b8 3c 58 float[3] [3]
39 b4 3c 58 39
b4 3c
3f40e914 62 98 b8 3c float 0.0225336 [0]
3f40e918 58 39 b4 3c float 0.022 [1]
3f40e91c 58 39 b4 3c float 0.022 [2]
3f40e920 60 d9 bb 3c 52 float[3] [4]
48 b2 3c 58 39
b4 3c
3f40e920 60 d9 bb 3c float 0.0229308 [0]
3f40e924 52 48 b2 3c float 0.021763 [1]
3f40e928 58 39 b4 3c float 0.022 [2]
3f40e92c 32 dd b5 3c 27 float[3] [5]
33 b9 3c 58 39
b4 3c
3f40e92c 32 dd b5 3c float 0.0222002 [0]
3f40e930 27 33 b9 3c float 0.0226074 [1]
3f40e934 58 39 b4 3c float 0.022 [2]
3f40e938 25 d0 bb 3c 58 float[3] [6]
39 b4 3c 49 36
b2 3c
3f40e938 25 d0 bb 3c float 0.0229264 [0]
3f40e93c 58 39 b4 3c float 0.022 [1]
3f40e940 49 36 b2 3c float 0.0217544 [2]
3f40e944 10 d6 b5 3c 58 float[3] [7]
39 b4 3c 76 87
b9 3c
3f40e944 10 d6 b5 3c float 0.0221968 [0]
3f40e948 58 39 b4 3c float 0.022 [1]
3f40e94c 76 87 b9 3c float 0.0226476 [2]
From trying to read the decompiled code, it seems like the calibration constants it chooses are based on how the phases are aligned on the input wires. I'm having a lot of trouble understanding what the code is doing, but it's around 401040dc in the ELF file, and looks something like this when decompiled:
if ((undefined *)(uint)DAT_3ffcc320 == (undefined *)0x0) {
pcStack44 = &DAT_3ffcc320;
puStack48 = (undefined *)(uint)DAT_3ffcc320;
line_00 = (undefined *)lineNo();
pcVar8 = &DAT_3f409e16;
log1(3,s_RECONNECT_3f40a774 + 7,&DAT_3f409e16,(int)line_00);
*pcStack44 = 1;
memw();
if (80.0 < fStack1356) {
if (fStack1344 < 40.0) {
fVar18 = 1.0 / (float)(ulonglong)_DAT_3ffcddc6;
if (ABS(fVar18 - 120.0) < 20.0) {
puVar4 = (undefined *)0x4;
}
else {
if (20.0 <= ABS(fVar18 - 240.0)) {
puVar4 = puStack48;
if (20.0 <= ABS(fVar18 - 180.0)) {
uVar3 = 0xc9;
goto LAB_401040b3;
}
}
else {
puVar4 = (undefined *)0x5;
}
}
}
else {
if (fStack1344 <= 80.0) goto LAB_401040c8;
puVar4 = (undefined *)0x1;
}
}
else {
LAB_401040c8:
if (40.0 <= fStack1356) {
LAB_4010413e:
uVar3 = 200;
LAB_401040b3:
*pcStack44 = uVar3;
memw();
goto LAB_40104015;
}
if (40.0 <= fStack1344) {
if (fStack1344 <= 80.0) goto LAB_4010413e;
fVar18 = 1.0 / (float)(ulonglong)_DAT_3ffcddc8;
if (20.0 <= ABS(fVar18 - 120.0)) {
if (20.0 <= ABS(fVar18 - 240.0)) {
if (20.0 <= ABS(fVar18 - 180.0)) {
uVar3 = 0xca;
goto LAB_401040b3;
}
puVar4 = (undefined *)0x2;
}
else {
puVar4 = (undefined *)0x7;
}
}
else {
puVar4 = (undefined *)0x6;
}
}
else {
puVar4 = (undefined *)0x3;
}
}
calibration(puVar4,s_RECONNECT_3f40a774 + 7,&DAT_3f409e16,line_00,s_RECONNECT_3f40a774 + 7);
}
puVar4 is the index into the array.
and like this when disassembled:
40103fa8 a2 a6 54 movi a10,0x654
40103fab 1a aa add.n a10,a10,a1
40103fad 88 0a l32i.n a8,a10,0x0
40103faf 0c 16 movi.n a6,0x1
40103fb1 62 48 00 s8i a6,a8=>DAT_3ffcc320,0x0
40103fb4 c0 20 00 memw
40103fb7 3d f0 nop.n
40103fb9 3d f0 nop.n
40103fbb 3d f0 nop.n
40103fbd 3d f0 nop.n
40103fbf 61 69 31 l32r a6,F_80.0 = 80.0
40103fc2 50 06 fa wfr f0,a6
40103fc5 62 a6 50 movi a6,0x650
40103fc8 1a 66 add.n a6,a6,a1
40103fca 23 01 4d lsi f2,a1,0x134
40103fcd 20 00 4b olt.s b0,f0,f2
40103fd0 a1 66 31 l32r a10,F_40.0 = 40.0
40103fd3 50 1a fa wfr f1,a10
40103fd6 92 26 00 l32i a9,a6,0x0
40103fd9 76 10 02 bt b0,LAB_40103fdf
40103fdc 06 3a 00 j LAB_401040c8
LAB_40103fdf XREF[1]: 40103fd9(j)
40103fdf 33 01 50 lsi f3,a1,0x140
40103fe2 10 03 4b olt.s b0,f3,f1
40103fe5 76 10 02 bt b0,LAB_40103feb
40103fe8 06 34 00 j LAB_401040bc
LAB_40103feb XREF[1]: 40103fe5(j)
40103feb 61 60 31 l32r a6,DAT_400d056c = 3FFCDDC4h
40103fee 62 16 01 l16ui a6,a6,offset DAT_3ffcddc6
40103ff1 00 06 da ufloat.s f0,a6,0x0
40103ff4 61 5f 31 l32r a6,F_120.0 = 120.0
40103ff7 50 26 fa wfr f2,a6
40103ffa 20 10 1a sub.s f1,f0,f2
40103ffd a1 5d 31 l32r a10,F_20.0 = 20.0
40104000 10 11 fa abs.s f1,f1
40104003 50 2a fa wfr f2,a10
40104006 20 01 4b olt.s b0,f1,f2
40104009 76 10 02 bt b0,LAB_4010400f
4010400c 06 1c 00 j LAB_40104080
LAB_4010400f XREF[1]: 40104009(j)
4010400f a2 a0 04 movi a10,0x4
LAB_40104012 XREF[7]: 40104094(j), 401040ac(j),
401040c4(j), 401040d9(j),
40104105(j), 4010411c(j),
40104134(j)
40104012 25 6a 05 call8 calibration = "CT"
= 1Bh
undefined calibration()
LAB_40104015 XREF[2]: 40103f80(j), 401040b9(j)
40104015 a5 9b 03 call8 provisioningStatusLessThan48Bytes bool provisioningStatusLessThan4
= "CT"
= 1Bh,"[0;32mI (%d) %s: Calculat
= 1Bh
40104018 16 2a 04 beqz a10,LAB_4010405e
4010401b e5 53 05 call8 getUxQueue = "CT"
void * getUxQueue(void)
= 1Bh
4010401e 16 ca 03 beqz a10,LAB_4010405e
40104021 65 1e 05 call8 uxQueueHasSpace bool uxQueueHasSpace(void)
= "CT"
= 1Bh
40104024 fc 6a bnez.n a10,LAB_4010405e
40104026 81 ed 30 l32r a8,->lineNo = 40086ec4
40104029 e0 08 00 callx8 a8=>lineNo = "CT"
= 1Bh
int lineNo(void)
4010402c c1 55 31 l32r a12,PTR_DAT_400d0580 = 3f409e43
4010402f dd 0a mov.n a13,a10
40104031 ed 03 mov.n a14,a3
40104033 bd 03 mov.n a11,a3
40104035 0c 3a movi.n a10,0x3
40104037 81 17 31 l32r a8,->log1 = 40086d88
4010403a e0 08 00 callx8 a8=>log1 = "CT"
int log1(int level, char * funcn
= 1Bh
4010403d ad 02 mov.n a10,a2
4010403f a5 cd 00 call8 V2Queue = "CT"
void * V2Queue(int param_1)
= 1Bh
40104042 62 05 05 l8ui a6,a5,offset DAT_3ffcc5fd
40104045 dc 56 bnez.n a6,LAB_4010405e
40104047 c2 a2 10 movi a12,0x210
4010404a 20 b2 20 mov a11,a2
4010404d a1 4d 31 l32r a10,DAT_400d0584 = 3FFCC350h
40104050 0c 16 movi.n a6,0x1
40104052 81 e5 30 l32r a8,->memcpy = 40091ab0
40104055 e0 08 00 callx8 a8=>memcpy = "CT"
void * memcpy(void * dest, void
40104058 62 45 05 s8i a6,a5,offset DAT_3ffcc5fd
4010405b c0 20 00 memw
LAB_4010405e XREF[4]: 40104018(j), 4010401e(j),
40104024(j), 40104045(j)
4010405e 20 a2 20 mov a10,a2
40104061 52 c7 3c addi a5,a7,0x3c
40104064 e5 83 00 call8 resetCalibrationData undefined4 * resetCalibrationDat
= "CT"
= 1Bh
40104067 57 34 05 bltu a4,a5,LAB_40104070
4010406a e5 c9 00 call8 loadAllCalibrations undefined4 loadAllCalibrations(u
= "CT"
4010406d 40 74 20 mov a7,a4
LAB_40104070 XREF[1]: 40104067(j)
40104070 b2 a0 00 movi a11,0x0
40104073 a2 a0 17 movi a10,0x17
40104076 a5 71 04 call8 FUN_40108790 = "CT"
undefined FUN_40108790()
40104079 40 64 20 mov a6,a4
4010407c 86 35 ff j LAB_40103d56
4010407f 00 ?? 00h
LAB_40104080 XREF[1]: 4010400c(j)
40104080 61 3e 31 l32r a6,F_240.0 = 240.0
40104083 50 36 fa wfr f3,a6
40104086 30 10 1a sub.s f1,f0,f3
40104089 10 11 fa abs.s f1,f1
4010408c 20 01 4b olt.s b0,f1,f2
4010408f 76 00 05 bf b0,LAB_40104098
40104092 0c 5a movi.n a10,0x5
40104094 86 de ff j LAB_40104012
40104097 00 ?? 00h
LAB_40104098 XREF[1]: 4010408f(j)
40104098 61 39 31 l32r a6,F_180.0 = 180.0
4010409b 50 16 fa wfr f1,a6
4010409e 10 00 1a sub.s f0,f0,f1
401040a1 10 00 fa abs.s f0,f0
401040a4 20 00 4b olt.s b0,f0,f2
401040a7 76 00 05 bf b0,LAB_401040b0
401040aa ad 09 mov.n a10,a9
401040ac 86 d8 ff j LAB_40104012
401040af 00 ?? 00h
LAB_401040b0 XREF[1]: 401040a7(j)
401040b0 62 af c9 movi a6,-0x37
LAB_401040b3 XREF[2]: 4010413b(j), 40104141(j)
401040b3 62 48 00 s8i a6,a8=>DAT_3ffcc320,0x0
401040b6 c0 20 00 memw
401040b9 06 d6 ff j LAB_40104015
LAB_401040bc XREF[1]: 40103fe8(j)
401040bc 30 00 4b olt.s b0,f0,f3
401040bf 76 00 05 bf b0,LAB_401040c8
401040c2 0c 1a movi.n a10,0x1
401040c4 86 d2 ff j LAB_40104012
401040c7 00 ?? 00h
LAB_401040c8 XREF[2]: 40103fdc(j), 401040bf(j)
401040c8 10 02 4b olt.s b0,f2,f1
401040cb 76 00 6f bf b0,LAB_4010413e
401040ce 23 01 50 lsi f2,a1,0x140
401040d1 10 02 4b olt.s b0,f2,f1
401040d4 76 00 04 bf b0,LAB_401040dc
401040d7 0c 3a movi.n a10,0x3
401040d9 46 cd ff j LAB_40104012
LAB_401040dc XREF[1]: 401040d4(j)
401040dc 20 00 4b olt.s b0,f0,f2
401040df 76 00 5b bf b0,LAB_4010413e
401040e2 61 22 31 l32r a6,DAT_400d056c = 3FFCDDC4h
401040e5 62 16 02 l16ui a6,a6,offset DAT_3ffcddc8
401040e8 00 06 da ufloat.s f0,a6,0x0
401040eb 61 21 31 l32r a6,F_120.0 = 120.0
401040ee 50 26 fa wfr f2,a6
401040f1 20 10 1a sub.s f1,f0,f2
401040f4 91 20 31 l32r a9,F_20.0 = 20.0
401040f7 10 11 fa abs.s f1,f1
401040fa 50 29 fa wfr f2,a9
401040fd 20 01 4b olt.s b0,f1,f2
40104100 76 00 04 bf b0,LAB_40104108
40104103 0c 6a movi.n a10,0x6
40104105 46 c2 ff j LAB_40104012
LAB_40104108 XREF[1]: 40104100(j)
40104108 61 1c 31 l32r a6,F_240.0 = 240.0
4010410b 50 36 fa wfr f3,a6
4010410e 30 10 1a sub.s f1,f0,f3
40104111 10 11 fa abs.s f1,f1
40104114 20 01 4b olt.s b0,f1,f2
40104117 76 00 05 bf b0,LAB_40104120
4010411a 0c 7a movi.n a10,0x7
4010411c 86 bc ff j LAB_40104012
4010411f 00 ?? 00h
LAB_40104120 XREF[1]: 40104117(j)
40104120 61 17 31 l32r a6,F_180.0 = 180.0
40104123 50 16 fa wfr f1,a6
40104126 10 00 1a sub.s f0,f0,f1
40104129 10 00 fa abs.s f0,f0
4010412c 20 00 4b olt.s b0,f0,f2
4010412f 76 00 05 bf b0,LAB_40104138
40104132 0c 2a movi.n a10,0x2
40104134 86 b6 ff j LAB_40104012
40104137 00 ?? 00h
LAB_40104138 XREF[1]: 4010412f(j)
40104138 62 af ca movi a6,-0x36
4010413b 06 dd ff j LAB_401040b3
LAB_4010413e XREF[2]: 401040cb(j), 401040df(j)
4010413e 62 af c8 movi a6,-0x38
40104141 86 db ff j LAB_401040b3
so there would be no way to get the sensor data to home assistant without some more development.
The native ESPHome API will be able to export the data to Home Assistant without the need for MQTT. Once the updates to the arduino libs are done, everything should work on the Arduino and esp-idf using their I2CDevice class. Right now, you should be able to test it without MQTT fine.
Also, there's still 2 things that I'm not sure what to do about them :
-
Calibration : I wonder if the calibration is really changing that much per CT. When doing my calculations, I was applying a single calibration value on every CT per phase, and it was returning the correct values that were sent out in the debug messages. Getting the ~0.022 value per phase seems to be easy enough since it's being printed to Serial, and could be captured before flashing ESPHome on the ESP32 quite easily.
-
"Current" : The thing is that I doubt that the "current" number is actually instantaneous current - I'm pretty sure it's some kind of current counter that allows the ESP32 to know how much current has passed in total across the CT. It would be a good idea to try to reverse this value to something we can use to properly calculate kWh values per CT. The Power values can't generate a reliable total energy consumption value alone. The way we could do that is by applying a constant load through a CT and try to find a correlation with the "current" value.
Also, I might try it out your fork @krconv on my bench today if I get the time.
TLDR because my message above was basically a dump of my thoughts:
I wonder if the calibration is really changing that much per CT
Calibration is not different per CT. Calibration is different depending on the phase order on the voltage sensing side in some way I do not understand.
Getting the ~0.022 value per phase seems to be easy enough since it's being printed to Serial, and could be captured before flashing ESPHome on the ESP32 quite easily.
Yup, this is probably the easiest way to do things. And there's only 6 combinations of wires, so the wire->calibration constants could be tested manually pretty easily.
The thing is that I doubt that the "current" number is actually instantaneous current
For some reason, I thought it was spot-on. It isn't quite, looking at a recent utility bill:
- I'm being billed for ~873kWh~ 790kWh
- Integrating & summing the two phases comes out to ~752.2kWh~ 782.1kWh
- Integrating & summing the various circuits comes out to ~752.5kWh~ 782.2kWh
- I'm being billed for 5.27kW peak consumption
- My charts show a 5.41kW peak consumption
But that still is very close, especially with the only calibration here being a global, per-voltage calibration that's pre-set from the factory and not device-specific.
edit: both misread my bill, and excluded a day in my report
- I'm being billed for 873kWh
- Integrating & summing the two phases comes out to 752.2kWh
- Integrating & summing the various circuits comes out to 752.5kWh
Yeah, that would be the kind of variation I would expect if it wasn't being counted correctly - Definitely something to investigate, since many people on the Emporia forum seems to say that the totals are really close to their utility bills (which makes sense since the 200A CT should rarely be below ~1A, their minimum for accurate readings)
Also, something interesting I noticed is that in the Pulseview dump from @flaviut, the intervals between data requests on the i2c bus aren't every second like I expected. They seem to be every ~240ms.
Something else that is intriguing is that the debug messages always seems to start by 0x03, and @krconv documented it as the message "version", but I'm seeing it with a value of 0x03 and 0x00 sometimes.
The first two bytes and the two bytes after seems to be some kind of counters, and, in the dump, the first one seem to alternate every message, while the second value is updated every 2 messages. E.g. msg1-0x03A2, msg2-0x00A2, msg3-0x0306, msg4-0x0006, msg5-0x0391, msg6-0x0091.
So I think It might indicate something else and be related to the "current" value, so we can sum it correctly.
- I'm being billed for ~873kWh~ 790kWh
- Integrating & summing the two phases comes out to ~752.2kWh~ 782.1kWh
- Integrating & summing the various circuits comes out to ~752.5kWh~ 782.2kWh
Do you have the current formula / procedure that you use to get from the current value to those power consumptions?
I completely ignore the current. I used this code to ingest the data into InfluxDB: https://github.com/flaviut/emporia-vue2-reversing/blob/master/parse_mqtt_dbg.py#L196-L242
And this query to process it:
import "math"
from(bucket: "home_automation")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "home_power")
|> filter(fn: (r) => r["_field"] == "power")
|> filter(fn: (r) => r["phase"] != "")
|> group(columns: ["_start", "_stop", "phase"])
|> integral(unit:1h)
|> group()
|> keep(columns: ["_value", "phase"])
I don't really understand why the integral() function takes a unit
Something else that is intriguing is that the debug messages always seems to start by 0x03, and @krconv documented it as the message "version", but I'm seeing it with a value of 0x03 and 0x00 sometimes.
Looking at the code, it seems like the existing code mostly ignores messages that do not have it set to 0x03:
if ((iVar1 == 0) && (pcVar6 = pcVar2 + 0x400, pcVar2[0x410] == '\x03')) {
pcVar7 = pcVar2 + 0x410;
pcVar8 = (char *)0x11c;
pvVar16 = memcpy(&uStack1664,pcVar7,0x11c);
FUN_40104f54(pvVar16,pcVar7,0x11c,line_00,pcVar9);
FUN_40109564(auStack104,pcVar7,0x11c,line_00,pcVar9);
pcVar7 = pcVar2;
FUN_4010432c(auStack332,pcVar2,0x11c,line_00,pcVar9);
pcVar6 = pcVar2;
}
uVar5 = FUN_40092448(0,pcVar6,pcVar8,line_00,pcVar9);
Of course, it's extremely likely I'm misunderstanding what is going on here.
Looking at the code, it seems like the existing code mostly ignores messages that do not have it set to
0x03:
I wouldn't be surprise, it would make sense that this would indicate to the ESP that the data is "ready". An 0x00 would indicate that the data is being updated, and then that would explain why the third and fourth bytes have a new value when we get a new 0x03.
I completely ignore the current. I used this code to ingest the data into InfluxDB
Looking at the code, you still seem to send the current into the influxDB. Would it be possible for you to dump the full timeseries for the period that match your known power usage (from your utility provider), so we could try and calculate the current formula that might give an even more precise power consumption value.
Here's the full data for the past 30 days: https://drive.google.com/file/d/14VMxbSIGjlCfecbEYWAQfevvbe-ccUAM/view?usp=sharing
Note: it extracts to 5.4GB of data.
| Day | kWh |
|---|---|
| Oct 27 | 17 |
| Oct 28 | 19 |
| Oct 29 | 15 |
| Oct 30 | 10 |
| Oct 31 | 21 |
| Nov 01 | 22 |
| Nov 02 | 22 |
| Nov 03 | 20 |
| Nov 04 | 19 |
| Nov 05 | 22 |
| Nov 06 | 21 |
| Nov 07 | 28 |
| Nov 08 | 19 |
| Nov 09 | 21 |
| Nov 10 | 26 |
| Nov 11 | 22 |
| Nov 12 | 21 |
| Nov 13 | 27 |
| Nov 14 | 18 |
| Nov 15 | 23 |
| Nov 16 | 31 |
| Nov 17 | 18 |
| Nov 18 | 18 |
| Nov 19 | 24 |
| Nov 20 | 19 |
| Nov 21 | 21 |
| Nov 22 | 22 |
| Nov 23 | 23 |
| Nov 24 | 10 |
| Nov 25 | 11 |
| Nov 26 | 23 |
| Nov 27 | 29 |
A couple observations I've had:
- ESPHome has a
total_daily_readingcomponent, which seems to be able to take in a sensor with power information and sum it up over the day - The Vue starts printing a message like (paraphrased) "hasn't uploaded data over wifi in 120 seconds, at 20% of in memory capacity"
The thing is that I doubt that the "current" number is actually instantaneous current
What if it is actually "the current passed since last reading"? i.e. the current that's passed since the last message with a 0x3 "version" header.
Theory: The ESP polls the sensor every ~240ms. Meanwhile, the sensor is on some clock where it'll calculate "instantaneous" (or ~2 second window) current draw, voltage, and calculate power consumption. Maybe the "version" acts as a "is new reading" flag, where reading it causes it to be cleared. To calculate total power draw accurately, every single sensor reading would need to be aggregated together, because missing one measurement would mean missing the power consumption in that ~2s window.
I'm still having a hard time understanding how AC power is calculated (i.e. I understand DC P=V*I, but the integrals for AC I might not understand right). But, if the "power" sensor readings really are power readings, and have a unit of W, then wouldn't we only need to sum those up over a time period to get Wh/KWh? That is where I think the total_daily_reading component might come in, because it a) calculates the time since last reading and b) multiplies the time delta by the instantaneous power reading and c) adds it to a total counter (which I think can be consumed by home assistant's energy dashboard)
I flashed some code to my Vue, and am successfully reading from I2C on the esp-idf framework. I also tested using Arduino (just to see what would happen without the bigger buffer), and the I2C call returns a 28 error code; couldn't find what that means though.
Here's the output being spit out from ESPHome
[VV][scheduler:152]: Running interval 'update' with interval=1000 last_execution=4294966081 (now=580)
[VV][i2c.idf:137]: 0x64 RX 030452A6C8FDFFFFDDFDFFFFE7FDFFFFC7B9FAFFC7B6FDFF18BEFDFFEA050000E5020000ED020000120500009EFFFFFFABFFFFFF69070000530100006D0100003FF8FFFFF0F9FFFFFFF9FFFFF861FBFFC63AFEFFC64BFEFFDF0FFBFFF6D4FDFFE7AFFDFF6FF70A00B32708002DEF07006B400B00EB330800EF1A08004C930A000AF507000DD20700CE58FBFF7128FEFF121CFEFF6781FAFFC2A8FDFF68B6FDFF3E45FBFF2A2AFEFF8763FEFFF861FBFFC63AFEFFC64BFEFFDF0FFBFFF6D4FDFFE7AFFDFF98E4F9FFAD17FDFFBB32FDFFAD0EFAFFEB42FDFFEF1AFDFFAA49FAFF4951FDFFEF35FDFF923D37332F33A
[VV][emporia_vue:031]: Raw Sensor Data: 03.04.52.A6.C8.FD.FF.FF.DD.FD.FF.FF.E7.FD.FF.FF.C7.B9.FA.FF.C7.B6.FD.FF.18.BE.FD.FF.EA.05.00.00.E5.02.00.00.ED.02.00.00.12.05.00.00.9E.FF.FF.FF.AB.FF.FF.FF.69.07.00.00.53.01.00.00.6D.01.00.00.3F.F8.FF.FF.F0.F9.FF.FF.FF.F9.FF.FF.F8.61.FB.FF.C6.3A.FE.FF.C6.4B.FE.FF.DF.0F.FB.FF.F6.D4.FD.FF.E7.AF.FD.FF.6F.F7.0A.00.B3.27.08.00.2D.EF.07.00.6B.40.0B.00.EB.33.08.00.EF.1A.08.00.4C.93.0A.00.0A.F5.07.00.0D.D2.07.00.CE.58.FB.FF.71.28.FE.FF.12.1C.FE.FF.67.81.FA.FF.C2.A8.FD
[V][sensor:062]: 'A': Received new state -0.592032
[D][sensor:113]: 'A': Sending state -0.59203 W with 0 decimals of accuracy
[V][sensor:062]: 'C': Received new state 1.578056
[D][sensor:113]: 'C': Sending state 1.57806 W with 0 decimals of accuracy
[V][sensor:062]: '1': Received new state 1.352917
[D][sensor:113]: '1': Sending state 1.35292 W with 0 decimals of accuracy
[V][sensor:062]: '2': Received new state 1.977260
[D][sensor:113]: '2': Sending state 1.97726 W with 0 decimals of accuracy
[V][sensor:062]: '3': Received new state -2.068984
[D][sensor:113]: '3': Sending state -2.06898 W with 0 decimals of accuracy
[V][component:186]: Component emporia_vue.sensor took a long time for an operation (0.18 s).
[V][component:187]: Components should block for at most 20-30ms.```
It seems like it's parsing it correctly according to the struct definition, and can successfully get the data from I2C. I compared the power values for a known load with ESPHome installed vs flashing the original software on there, and the values seemed close. I noticed with ESPHome, the values fluctuate between -2 W and 2 W though, where it is exactly zero with the original software; I wonder if the original software filters out readings below 2 watts or something like that.
This is the config I'm testing with:
Config
substitutions:
device_name: home-power-monitor
friendly_name: Home Power Monitor
assigned_ip_address: "192.168.86.223"
esphome:
name: ${device_name}
esp32:
board: esp-wrover-kit
framework:
type: esp-idf
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
use_address: ${assigned_ip_address}
ota:
logger:
level: VERY_VERBOSE
i2c:
sensor:
- platform: emporia_vue
update_interval: 1s
phases:
- id: phase_a
input: BLACK
power:
- name: "A"
phase_id: phase_a
input: "A"
- name: "C"
phase_id: phase_a
input: "C"
- name: "1"
phase_id: phase_a
input: "1"
- name: "2"
phase_id: phase_a
input: "2"
- name: "3"
phase_id: phase_a
input: "3"
external_components:
- source: github://krconv/esphome@add-emporia-vue
components: [emporia_vue]
refresh: 1min
Wow, good job on the code!
Here are some things I found while looking at the code and log output :
-
The update loop seems to take ~180ms, which is way too much. But looking at the default I2C baud rate (which I believe is what's being used by your component), it's 50kHz, which is less than the default 100kHz that the Emporia firmware was using. 284 bytes at 50kHz takes around 70ms to read, vs ~35ms @ 100kHz. We should override the i2c default to at least 100kHz. Per this doc, it looks like we should be able to get away with even faster baud rates, like 400kHz (~9ms), shaving precious execution time on
i2c_device::read(). Keep in mind that the baud rate is set by the master (the ESP32 in this case) so the slave should follow whatever rate we set, if it can process the signal at this rate. -
Because of the way the interaction with the co-MCU works, I don't think we should tie it to the sensor refresh
update()loop. We should create a separate sub-task/routine to allow precise timings on the i2c reads (maybe similar to the Emporia firmware with it's 240ms pull interval), allowing us to properly collect the co-MCU data and calculate the power usage. Theupdate()routine would then just request the most up-to-date data or the accuratecumulative energyvalues from that subroutine. Also, when tying the request to theupdate()loop, we risk hitting a message that starts with0x00, which we should ignore. -
Your calibrated power doesn't seem to factor in the difference in factor between CT 0-2 (5.5) and CT 3-18 (22) when checking this line : https://github.com/krconv/esphome/blob/bcce62b67517046ad2c2f800d17d0c817a78be43/esphome/components/emporia_vue/emporia_vue.cpp#L60
-
Your calibrated power also seems to use a single calibration value. We should have an array of 3 calibration values (1 per phase) and apply the calibration value of the phase the CT is configured with.
-
I'm not sure if the console was overflowing, but both your serial print from
i2c.idfandemporia_vueare missing some bytes. It sure does like it, since they both stopped at the same character count, and at different hex bytes.
Edit: Here's a good example on how we could start a subtask on a separate thread that would handle the i2c reads and power value calculations in a non loop-blocking way : https://github.com/esphome/esphome/blob/939fb313df622d3fabc14ea837e296c7aa5ec0ec/esphome/components/esp32_camera/esp32_camera.cpp#L35
I also tested using Arduino (just to see what would happen without the bigger buffer), and the I2C call returns a 28 error code
@krconv I'm 99% sure that it's this here : https://github.com/esphome/esphome/blob/939fb313df622d3fabc14ea837e296c7aa5ec0ec/esphome/components/i2c/i2c_bus_arduino.cpp#L75
It's just telling you that the amount of byte read is not equal to the requested len, which would be accurate since we're overflowing the uint8_t.
@krconv Would it be possible for you to give us write access to your esphome fork so we can all work on the same repo?
This is very exciting! Nice job @krconv, I've also read all your code & it looks very clean!
With regards to the calibration, I thought you were planning to use ESPHome's built-in linear calibration? Is this change a user-friendliness change, or does using the built-in thing break something else?
Thanks for the review on the code; really helpful, I'm still trying to understand how to work with ESPHome. Just gave you both access, feel free to push directly or PR as you desire. The calibration code I had in there was mainly for local testing to see if I could get anything consistent with real values (I was uploading to GitHub and downloading it onto my Home Assistant machine to test); I added a few more TODOs to mark things that aren't done yet too.
I tried out 400kHz, which seems to work sometimes, but also sometimes times out ([20:59:43][VV][i2c.idf:119]: RX from 64 failed: timeout). Tried 200kHz, which seems to be reliable, and tried out 800kHz to see what would happen, and that causes I2C timeouts on every read.
For calibration; I'm still wrapping my head around it, and I don't feel strongly either way. I think using some "fake" calibration data to at least get the numbers to seem more understandable might be good for user friendly-ness, even if they aren't accurate, and could include instructions for how to calibrate further. If getting the phase calibrations from logs works as we expect, I think we should also give an option to plug those in directly as well to help avoid manual calibration. I still don't trust the readings, and I kinda think we're gonna run into something that encourages us to use the other sensor data that we aren't currently reading (maybe the counter in the last byte of the header or the current data); I feel like there's so much going on in the logs, and other pins transmitting data on the board that reading the sensor data directly feels like we're cheating somehow.
I like @Maelstrom96 idea to control the how often we poll from the MCU ourselves, and let how often we actually publish the value be controlled by config. Tonight I tested calibrating one of my circuits to read accurate values (using calibrate_linear), and then adjusting how often we read from the sensor to see how it affects the readings. It seems like when we read from the sensor, we only get the last ~1s of data, so if we wait too long then we miss data. For example, I set the update interval to 5s, and turned the stove on for 4 seconds, and turned it off right before the reading, and the sensor just returned a reading of 0W, instead of an average over the 5 seconds. So it seems like we'll need to implement that averaging mechanism.
It is really cool to see how fast the values come into Home Assistant. It's realtime, instead of the ~5 second delay with the Emporia app. I get a kick out of turning the stove on and off just to watch the numbers bounce back and forth 😆
It seems like when we read from the sensor, we only get the last ~1s of data, so if we wait too long then we miss data.
So my latest theory is that the co-MCU is averaging the last ~480ms of power consumption into the power readings values. So in order to get accurate total energy counts, we might need to compile the values using all the message data as soon as it's updated.
I don't have the time to test it right now, but here are some changes I made to the codebase if you want to test them @krconv : https://github.com/Maelstrom96/esphome/tree/add-emporia-vue2
Edit: Just thought about it and you might not be able to test it using the "External_Component" configuration since I made changes to the esphome/const.py file.
Edit2: Changed the code so it could be correctly imported for testing. Also, you will have to rename the "power" configuration to "ct".
I can take a look tonight! I've been playing around with putting the read task in the loop(), interested to see how the scheduled task works. BTW, I noticed that averaging a few reading together makes the readings around 0W more stable which is encouraging.
Got my the two 400A clamps on the main power cables, and running that for a day to see if our calculations are close to my power company's readings 🤞
@krconv Not sure if it's just something with my setup, but when using the esp-idf framework instead of the default Arduino one, the wifi just doesn't want to connect, and the serial/UART output is all over the place. I've tried installing the code without the emporia_vue module and by cleaning the build files to make sure that wasn't playing a role, and it's still happening with a pretty bare configuration file. Are you also seeing this? I'm using the latest ESPHome 2021.11.04.
[V][wifi_esp32:434]: tcpip_adapter_dhcpc_stop failed: ESP_ERR_ESP_NETIF_DHCP_ALREADY_STOPPED
[E][wifi:291]: wifi_sta_connect_ failed!
[D][wifi:370]: Starting scan...
[V][wifi_esp32:647]: Event: WiFi Scan Done status=0 number=18 scan_id=177
[D][wifi:385]: Found networks:
[I][wifi:429]: - 'AJ - IOT' (FA:92:BF:5D:EF:A6) ▂▄▆█
[D][wifi:430]: Channel: 5
[D][wifi:431]: RSSI: -38 dB
[I][wifi:429]: - 'AJ - IOT' (FA:92:BF:5D:F4:37) ▂▄▆█
[D][wifi:430]: Channel: 5
[D][wifi:431]: RSSI: -52 dB
[D][wifi:434]: - 'AJ - 2.4Ghz' (FE:92:BF:5D:EF:A6) ▂▄▆█
[D][wifi:434]: - 'AJ' (F4:92:BF:5D:EF:A6) ▂▄▆█
[D][wifi:434]: - '' (02:92:BF:5D:EF:A6) ▂▄▆█
[D][wifi:434]: - 'AJ' (F4:92:BF:5D:F4:37) ▂▄▆█
[D][wifi:434]: - 'AJ - 2.4Ghz' (FE:92:BF:5D:F4:37) ▂▄▆█
[D][wifi:434]: - '' (02:92:BF:5D:F4:37) ▂▄▆█
[D][wifi:434]: - 'JLWIFI' (90:AA:C3:37:0F:58) ▂▄▆█
[D][wifi:434]: - 'yeelink-light-color2_miapC64C' (04:CF:8C:7C:C6:4C) ▂▄▆█
[D][wifi:434]: - 'BELL641' (48:29:52:FA:48:7E) ▂▄▆█
[D][wifi:434]: - 'yeelink-light-color2_miapCAC7' (04:CF:8C:7C:CA:C7) ▂▄▆█
[D][wifi:434]: - 'oxio-7410' (E8:2C:6D:50:74:14) ▂▄▆█
[D][wifi:434]: - 'WiFiGilles' (10:33:BF:E6:25:7E) ▂▄▆█
[D][wifi:434]: - '' (10:33:BF:E6:25:81) ▂▄▆█
[D][wifi:434]: - 'oxio-7410' (3C:90:66:F8:E8:64) ▂▄▆█
[D][wifi:434]: - 'HP-Print-36-ENVY 4500 series' (64:51:06:EC:E1:36) ▂▄▆█
[D][wifi:434]: - 'BELL872' (C0:3C:04:1B:FA:86) ▂▄▆█
[I][wifi:245]: WiFi Connecting to 'AJ - IOT'...
[V][wifi:247]: Connection Params:
[V][wifi:248]: SSID: 'AJ - IOT'
[V][wifi:251]: BSSID: FA:92:BF:5D:EF:A6
[V][wifi:271]: Password: NOPE
[V][wifi:276]: Channel: 5
[V][wifi:283]: Manual IP: Static IP=172.16.10.4 Gateway=172.16.10.1 Subnet=255.255.255.0 DNS1=0.0.0.0 DNS2=0.0.0.0
[V][wifi:287]: Hidden: NO
[V][wifi_esp32:434]: tcpip_adapter_dhcpc_stop failed: ESP_ERR_ESP_NETIF_DHCP_ALREADY_STOPPED
[E][wifi:291]: wifi_sta_connect_ failed!
[D][wifi:370]: Starting scan...
[V][wifi_esp32:647]: Event: WiFi Scan Done status=0 number=18 scan_id=178
[D][wifi:385]: Found networks:
[I][wifi:429]: - 'AJ - IOT' (FA:92:BF:5D:F4:37) ▂▄▆█
[D][wifi:430]: Channel: 5
[D][wifi:431]: RSSI: -53 dB
[I][wifi:429]: - 'AJ - IOT' (FA:92:BF:5D:EF:A6) ▂▄▆█
[D][wifi:430]: Channel: 5
[D][wifi:431]: RSSI: -38 dB
[D][wifi:434]: - 'AJ' (F4:92:BF:5D:EF:A6) ▂▄▆█
[D][wifi:434]: - 'AJ - 2.4Ghz' (FE:92:BF:5D:EF:A6) ▂▄▆█
[D][wifi:434]: - '' (02:92:BF:5D:EF:A6) ▂▄▆█
[D][wifi:434]: - ''
Flashing the same config with Arduino seems to work fine, but wouldn't work OK with our i2c requirements.
I did hit an issue with connecting to the board remotely once (I've been uploading OTA), but I'm not sure if it was WiFi or not. Whatever it was, cutting power and restarting fixed it for me. Here is the full minimal config/console that I tried:
Config
esphome:
name: home-power-monitor
esp32:
board: esp-wrover-kit
framework:
type: esp-idf
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
use_address: "192.168.86.223"
ota:
logger:
level: DEBUG
api:
Output
[19:52:00][I][app:099]: ESPHome version 2021.11.4 compiled on Dec 1 2021, 19:50:35 [19:52:00][C][wifi:488]: WiFi: [19:52:00][C][wifi:350]: Local MAC: 40:F5:20:6A:26:68 [19:52:00][C][wifi:351]: SSID: [redacted] [19:52:00][C][wifi:352]: IP Address: 192.168.86.223 [19:52:00][C][wifi:354]: BSSID: [redacted] [19:52:00][C][wifi:355]: Hostname: 'home-power-monitor' [19:52:00][C][wifi:357]: Signal strength: -30 dB ▂▄▆█ [19:52:00][C][wifi:361]: Channel: 11 [19:52:00][C][wifi:362]: Subnet: 255.255.255.0 [19:52:00][C][wifi:363]: Gateway: 192.168.86.1 [19:52:00][C][wifi:364]: DNS1: 192.168.86.1 [19:52:00][C][wifi:365]: DNS2: 0.0.0.0 [19:52:00][C][logger:233]: Logger: [19:52:00][C][logger:234]: Level: DEBUG [19:52:00][C][logger:235]: Log Baud Rate: 115200 [19:52:00][C][logger:236]: Hardware UART: UART0 [19:52:00][C][ota:082]: Over-The-Air Updates: [19:52:00][C][ota:083]: Address: 192.168.86.223:3232 [19:52:00][C][api:134]: API Server: [19:52:00][C][api:135]: Address: 192.168.86.223:6053 [19:52:00][C][api:139]: Using noise encryption: NO [19:52:00][C][mdns:084]: mDNS: [19:52:00][C][mdns:085]: Hostname: home-power-monitor
Good call on deleting the build dir; I've noticed that some of the flags that change for the esp-idf library don't properly recompile all of the libraries.
I tested out that branch on my Vue, and the repeating task seemed to work first try. My notes
- Publishing zero values; the
sensor.pyto_codeis missing theset_calibrationto put the config value into C++ - For some reason, OTA updates wouldn't work; I had to put it in safe mode to flash OTA again, I wonder if the task is taking too much time from the Wifi or OTA components
- Because it is publishing so frequently to home assistant, the UI for the sensor history has a hard time loading; I think giving a way to configure how often the values are update would help
I updated my branch here too; I copied some of your code (I'll figure out how to get the commits to show credit correctly later, committed them all as me to avoid the rebase/cherry-pick for now). Two notable difference
- left the main component as a polling component; the power sensors read and average readings together between polls
- using the loop for "async tasks" instead of RTOS; I think it's more straightforward (I've been looking at the
ct_clampcomponent as an example)
I'm still waiting for the latest power readings from my power company; if they seem accurate, how would you guys feel about me opening a PR to the main repo? It seems pretty common for PRs to be open for a while with ESPHome and to make changes/give feedback, and I think it'll be good for visibility and to make it easier to for us and others review code with inline comments
So regarding my issue with the wifi component, it was a problem when using the manual_ip setting. I've submitted a PR to fix it and it has been merged.
The main mentality with esphome components seems to be to leave the data as raw as possible and change the behavior of the sensors using filters. For you data rate issue, you should be able to just setup a throttle filter. Also, a reason that we would want all those values is so that our total_daily_energy sensors are as accurate as possible. Even if you apply filters, the energy component will get the ~500ms state update and it will be able to calculate the most precise power consumption possible.
For the task I made, it should not be 100% blocking since other tasks should run when we're in vTaskDelay. Still need to investigate why ota wasn't working for you. I will be testing it locally.
A big problem I see with the way you do the loop is that the blocking/execution time will be way to high. There's also potential for even higher block time with this->read in cases where the read times out for whatever reason. I haven't found a way to change the default timeout of 1s yet.
Cool!
For the polling component vs publish on every reading; I don't think we are loosing any accuracy in our data by not publishing every value, they are still all accounted for in the average. People would need to add a filter to each of their CT sensors to reduce the frequency if they want the power data in Home Assistant (i.e. not just energy usage but current power load as well), or otherwise risk publishing too many updates. I think setting a single update_interval on the component is more intuitive, even if it isn't what is going on under the hoods; anyone could still set update interval to ~510ms if they want to speed up the readings.
I notice the priority on the RTOS task is set to 0; maybe reducing it would help (just a guess)? FWIW, the read loop is taking 23ms to read from I2c, which I think is ok given that this is working with specialized hardware and not intended for general purpose use. Aren't we taking the same amount of time with RTOS as well, or do they run on different cores? I'd argue giving ESP control over the loop would provide easier debugging, even if we occasionally get warnings that the loop took too long.
Got the most recent readings from my power company, and the values look pretty close to accurate as is! Mine were about 2% below the power company's readings, which I hope is only due to me not calibrating the phases exactly (using the default of 0.022 because I didn't get the calibration data before flashing). The missing sections are times of the day I was testing the Vue and readings weren't accurate
Regarding the task, the whole point was to offload the i2c read to the second CPU core, so it is running on the other core (cpu1). Also, I've just checked to validate the information, and the lower the priority number, the lower the priority is, so 0 should be the lowest one, which is the same priority as the idle task (see https://www.freertos.org/RTOS-task-priority.html)
@krconv If you were using the dev branch, this might explain why your OTA wasn't working : https://github.com/esphome/esphome/pull/2852
I was flashing using the released version; trying to get a few things done this morning:
- Rebase my non-controversial changes onto your branch; I made a few naming (hopefully) improvements, and added logging (done: https://github.com/Maelstrom96/esphome/pull/2)
- Get all the changes working and test on my Vue
- Make a separate PR to your fork with more discussion on the RTOS task, and another idea I just got for configuration (similar to the voltage sensor you added, which I think is a great idea and I think we should apply to power and current as well)
Just a FYI : https://github.com/esphome/esphome/pull/2871
I think we should have this merged before implementing new sensors and features (like calibration mechanism). The maintainers seem to prefer smaller PRs so it should be easier to merge other features afterwards.
Thanks for the hard work, I've been reading the progress here, and it's incredible what the three of you have got going! I picked up an emporia vue last week and I am yet to install it. I don't have anything useful to contribute towards the reverse engineering efforts, but I can help beta (alpha?) test your changes, if you are looking for volunteers.
I second what nightweyr said! I've been following along with this too, but I'm not sure that I have the skills to help with reverse engineering, but I could see what I could do.
I also just ordered an emporia vue and plan on holding off on installing it until there's hopefully something I can flash to it prior to it connecting anywhere else.
Lmk if there's anything that needs to be done next that I might be able to help with!
Hey @nightweyr, @williamobrien, glad to have you here!
I've prepared some instructions on getting this thing installed here: https://gist.github.com/flaviut/93a1212c7b165c7674693a45ad52c512
I'd appreciate hearing any questions & comments you may have, so that I can improve the document!
@flaviut Thanks for the detailed instructions, they are much nicer than I was expecting.
I am running into one tiny problem. If I set this as per the doc:
esp32:
board: d-duino-32
framework:
type: esp-idf
This block complains that This feature is only available with frameworks ['arduino']
esphome:
name: sub-panel
If I set framework to arduino, esphome complains that the emporia_vue platform is only available for framework: esp-idf.
I am on esphome-stable 2021.12.1. I do recall @Maelstrom96 submitting a few patches to core for a few bugfixes. Should I switch to the dev branch for this? I'll spin up a docker container on another host to test this - my main esphome is running on my home assistant box, and I am wary of switching that over to dev.
I see that your board had headers soldered on the breakout pins for serial, power and GPIO0. I have two of them that I purchased last month which didn't have them. I don't think this should be a blocker for anyone willing to open their vue and flash alternate firmware.
@nightweyr I'm also on 2021.12.1, and I don't get that error when I try and compile it. Would you mind posting the whole error message?
I see that your board had headers soldered on the breakout pins for serial, power and GPIO0. I have two of them that I purchased last month which didn't have them. I don't think this should be a blocker for anyone willing to open their vue and flash alternate firmware.
Yup, I soldered them on there since development is a lot longer than using something pre-developed. Definitely shouldn't be a problem for people who don't have a soldering iron, since they can just hold the jumpers in place for the minutes that this takes.
@nightweyr I'm also on 2021.12.1, and I don't get that error when I try and compile it. Would you mind posting the whole error message?
Same here, might be a good idea to try and still try and compile the code.
@flaviut Also, in your guide, I would change the backup command to esptool.py -b 921600 read_flash 0 0x800000 flash_contents.bin instead. This will backup the whole flash instead of just where the runtime code resides. Also, setting the baud rate will significantly speed up the read_flash operation.
@Maelstrom96 My understanding is that the ESP32-WROVER-B has 4 MiB of flash, is there something else I'm missing here? Good idea on the baud rate though!
@flaviut The ESP32-WROVER-B doesn't have onboard flash and instead relies on flash is added by the integrator. In this case, in the logs, it was telling me that 8MB (64MBit) flash was detected when doing a read or flash.
I (40) boot: Enabling RNG early entropy source...
I (44) boot: SPI Speed : 40MHz
I (48) boot: SPI Mode : DIO
I (52) boot: SPI Flash Size : 8MB
I (670) psram: This chip is ESP32-D0WD
Huh, neat, guess that's the case. To be nitpicky, unless I'm missing something else, the datasheet specifies 4MiB flash unless it is a custom order (sec 1), but I failed to consider that they're using external flash.
Thanks for the correction, I've fixed it!
Oh, I wasn't sure why I thought that there wasn't any internal flash until I check the spec sheet for the ESP32-D0DW, which specifies that there is no integrated flash. It is true that this particular ESP32 package has an "external flash" on the same board, and I guess they custom ordered them to have 8MB.
@nightweyr I'm also on 2021.12.1, and I don't get that error when I try and compile it. Would you mind posting the whole error message? With my usual template:
esphome:
name: sub-panel
external_components:
- source: github://Maelstrom96/esphome@add-emporia-vue2
components: [ emporia_vue ]
esp32:
board: d-duino-32
framework:
type: esp-idf
version: recommended
# Common section begin
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
domain: !secret wifi_domain
# Enable fallback hotspot (captive portal) in case wifi connection fails
ap:
ssid: "$devicename Fallback Hotspot"
password: !secret fallback_pw
# captive_portal:
# Enable logging
logger:
web_server:
ota: false
esp32_improv:
authorizer: none
improv_serial:
# remove # below to enable ble tracking
#esp32_ble_tracker:
# Enable Home Assistant API
api:
password: !secret api_pw
ota:
password: !secret ota_pw
# Common section end
I get this error:
root@15ef4d2f-esphome:/# esphome -v compile /config/esphome/sub-panel.yaml
INFO Reading configuration /config/esphome/sub-panel.yaml...
Failed config
json: None
This feature is only available with frameworks ['arduino'].
{}
async_tcp: None
This feature is only available with frameworks ['arduino'].
{}
If I comment out the framework part of the yaml, I get this (which, to be clear, should be expected):
root@15ef4d2f-esphome:/# esphome -v compile /config/esphome/sub-panel.yaml
INFO Reading configuration /config/esphome/sub-panel.yaml...
Failed config
sensor.emporia_vue: [source /config/esphome/sub-panel.yaml:58]
This feature is only available with frameworks ['esp-idf'].
platform: emporia_vue
i2c_id: i2c_a
phases:
- id: phase_a
input: BLACK
voltage:
name: Phase A Voltage
- id: phase_b
input: RED
voltage:
name: Phase B Voltage
ct_clamps:
- name: A
phase_id: phase_a
input: A
- name: B
phase_id: phase_b
input: B
- name: 1
phase_id: phase_a
input: 1
- name: 2
phase_id: phase_a
input: 2
- name: 3
phase_id: phase_a
input: 3
Yup, I soldered them on there since development is a lot longer than using something pre-developed. Definitely shouldn't be a problem for people who don't have a soldering iron, since they can just hold the jumpers in place for the minutes that this takes.
Ha, I should have guessed that. I do have a few alligator clips to dupont jumpers at home. Let me try searching tor where they are, and for scientific reasons, let me see if I can clip them on steadily enough to complete the flash.
@nightweyr
Just tested your exact config and I got the exact same validation errors as you. Upon checking it and seeing if removing components would remove the validation warning, I found that removing the web_server configuration was removing the validation error. Might be worth it to open a ESPHome issue since the validation error message and in the UI is very misleading in this particular case.
~~Mind me asking why you had that configuration~~
web_server:
ota: false
~~while still having set~~
ota:
password: !secret ota_pw
Edit: Just checked and that web_server setting as nothing to do with normal OTA, so nvm my question. But I think web_server is not supported with esp_idf. Good news tho, we should be able to use the Arduino library with the emporia_vue component soon since we were able to have the library fixed for our use case. I will have to add some code and test that everything works, but it should come eventually.
Duh, thanks! Sorry for the false alarm. I took out the webserver section and was able to compile the binary without any issues. The reason I didn't suspect it was because of the esphome error message, I'll submit a bug report there after testing a few more conditions. It's too late here to continue, but I hope to wrap this up this week and install it during the weekend.
Wow, those are some great instructions - thanks for putting that together! I'm just getting a chance to look at this right now and hopefully, in the next couple of days I'll have some time to give this a try.
One quick question - looking at the configuration, where would we enter the multiplier? For example, for a double pole breaker, would it be something like this?
- name: "2"
phase_id: phase_a
input: "2"
multiplier: "2"
Good question @williamobrien, I'll need to update the guide.
But there's https://esphome.io/components/sensor/index.html#calibrate-linear and I think that should take care of it!
Good question @williamobrien, I'll need to update the guide.
But there's https://esphome.io/components/sensor/index.html#calibrate-linear and I think that should take care of it!
Kind of overkill to use calibrate-linear for that. @williamobrien, you can double the value like this :
- name: "2"
phase_id: phase_a
input: "2"
filters:
- multiply: 2.0
Quick question - were your header holes covered/filled with some kind of gel-like/stickyish substance? I'm assuming I can just remove it/poke through it, but wanted to see if that was how they all are.
When I first opened the device, I also noticed that there was something on the PCB, so you're not alone. Most likely a dielectric gel to protect the high voltage circuitry.
Both of my vue2s also had that goop. It smelt like left over rosin. It burns off very easily with a solder iron. Also, 115200 was the highest baud rate I could set for read_flash. Any higher and I got error messages like this: A fatal error occurred: Corrupt data, expected 0x1000 bytes but received 0xf88 bytes
I am using an FTDI clone that I had for well over a decade, so I suspect this problem is on my end. Flashing went through without a hitch, but I have to hook it up this weekend to test the device for real.
I didn't have any goop at all. Do you have a picture of it? It'd be helpful in improving the instructions.
I have issues communicating with the module. I'm receiving following error: Failed to connect to Espressif device: No serial data received. Anyone has a suggestion? :)
Flip the tx and rx? Make sure you have good contact between the jumper wires and the board?
I have issues communicating with the module. I'm receiving following error: Failed to connect to Espressif device: No serial data received. Anyone has a suggestion? :)
Did you make sure to follow the steps to put the module in "download mode" using the reset and IO0 pin?
I tried few different combinations. What is exact procedure that works for you?
Flip the tx and rx? Make sure you have good contact between the jumper wires and the board?
Double check it, even tried changing it on purpose, still no luck. My assumption is that it is not in download mode. Is there an led indicating it is in right mode or any other indicator? Thanks
Flip the tx and rx? Make sure you have good contact between the jumper wires and the board?
Double check it, even tried changing it on purpose, still no luck. My assumption is that it is not in download mode. Is there an led indicating it is in right mode or any other indicator? Thanks
The way I do it is by putting a jumper wire on both the IO0 and EN (which is the reset pin) with a male end, then you put both the EN and IO0 jumper on the ESP32 shield, inorder to ground both pins. Then, remove the EN jumper from the ESP shield while keeping the IO0 jumper on the shield for at least 1-2 seconds before removing it.
You can validate that everything works correctly by opening a serial console while doing this with a baud rate of 115200 and then proceeding with the procedure as described above. If all goes well, it should output in the serial console that it's in download mode. Checking the serial log will also help you confirm that your TX/RX pins are in the right order.
I'm not sure if this is a test mode thing, but on my system I get a long beep from the buzzer if it is in "execute" mode. No beep indicates that it is in programming mode.
On Fri, Dec 24, 2021, 15:31 Care4Home @.***> wrote:
Flip the tx and rx? Make sure you have good contact between the jumper wires and the board?
Double check it, even tried changing it on purpose, still no luck. My assumption is that it is not in download mode. Is there an led indicating it is in right mode or any other indicator? Thanks
— Reply to this email directly, view it on GitHub https://github.com/flaviut/emporia-vue2-reversing/issues/1#issuecomment-1000927306, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHY2HPDGSTFQSPHRTENBVDUSTKDPANCNFSM5IRAJONA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID: @.***>
I didn't have any goop at all. Do you have a picture of it? It'd be helpful in improving the instructions.
yes, here's a picture of the goop on the header pins:
I've updated the guide to address what I think is flux on the pins, and to use the remove-freertos-task branch, which I've found to be more stable in my use over the past week. The new revision of the guide is still here.
If you would like to see/use the old version, please take a look here.
I'd like to reach out to the interested communities[1][2][3] about this in a couple weeks or so, so I'd really appreciate criticism & compliments on the guide. Or to be told to hold off on this for a while longer :)
A video would be great. I haven’t done this yet but the instructions seem clear until I get to the remove any jumper from the usb. Does that mean remove all jumpers or just randomly pick one?
Mine looks like this 
, with a jumper between either VCC & 5V, or VCC & 3V3.
It ought to look like this, with no jumper attached to VCC:
Ok that makes sense. I haven’t purchased a usb to serial adapter yet. I wanted to make sure I could understand it all before spending money to dive into it.
Exciting little project... I received my Vue last week and followed flaviut's "Reverse-engineering" guide to get it talking to mosquitto / node-red running on a Raspberry Pi via MQTT. The parse_mqtt_dbg.py is pretty straight forward and outlines how to process the MQTT messages into an InfluxDB, which I am looking to reimplement in node-red. I notice that it's over 4 months old. Is there any newer code / documentation available which outlines the processing of the MQTT messages? Or is that pretty much all there is to it???
The circuit breaker on my hot tub trips a couple times each winter and I almost lost the hot tub last year due to the breaker tripping (it was well below freezing and the power was off for a few days). By monitoring the power usage I can get an alert as soon as the breaker trips again... Thanks for your hard work and dedication to this project. It really opens up this sweet little device as a powerful home automation tool.
The parse_mqtt_dbg.py is pretty straight forward and outlines how to process the MQTT messages into an InfluxDB, which I am looking to reimplement in node-red. I notice that it's over 4 months old. Is there any newer code / documentation available which outlines the processing of the MQTT messages?
That's all there is to it. process_message() returns a Vue2Reading, which you can use however you want. Only a few of the lines are MQTT-specific.
The reason there hasn't been any more work on that is that running fully open-source firmware is less janky (doesn't require specific IP addresses, working with copywrited firmware, or any limits on the wifi password). So all effort has gone into the FOSS firmware, through ESPHome.
The circuit breaker on my hot tub trips a couple times each winter and I almost lost the hot tub last year due to the breaker tripping (it was well below freezing and the power was off for a few days).
Wow! That's an awesome use, and I'm glad your hot tub survived!
Maybe you could use another esp32, duct tape, and one of these to make yourself a self resetting breaker :grin:
lol... I'll see what I can do about the automated breaker resetter...
Thanks again... You guys rock!
I've finally taken the plunge and ordered mine yesterday after finding this thread. Just wanted to say thanks for all the hard work, brilliant guide and give a quick bit of feedback on how things went for me. The unit and PCB I got look exactly like the pictures in the guide, I had the extra flux on the solder pads which was fairly easy to remove. My solder pads were full so I drilled them out with a tiny Dremel drill and soldered in headers. Everything from thereon was surprisingly straightforward. I've connected the cables from the unit's supply tail into a 13amp plug (black - Live, all other = Neutral), given that my consumer unit doesn't have enough spare room or a free breaker I'm going to mount the Emporia next to the consumer unit.
-
Backup from Ubuntu with a basic USB adapter: esptool.py version esptool.py v2.8 esptool.py -b 921600 read_flash 0 0x800000 flash_contents.bin esptool.py v2.8 Found 1 serial ports Serial port /dev/ttyUSB0 Connecting...... Detecting chip type... ESP32 Chip is ESP32D0WDQ5 (revision 1) Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None Crystal is 40MHz MAC: 40:f5:20:88:03:d8 Uploading stub... Running stub... Stub running... Changing baud rate to 921600 Changed. 8388608 (100 %) Read 8388608 bytes at 0x0 in 153.4 seconds (437.5 kbit/s)... Hard resetting via RTS pin...
-
Over to HA, create a new ESP device, copy over the example code from the "Setting up" guide and compile to download.
-
Attach USB adapter under MacOS, connect device and flash the .bin with ESPHome Flasher.
-
Quick look into ESPHome on HA and the device has connected ok, follow the usual "new device found" dialog and all the sensors come through.
I've hooked up one of the big CT clamps (that's all we need here in the UK for a normal domestic supply) and one of the little CT clamps on channel one. Readings come through ok and look correct, so far everything seems stable and I'm going to watch it for a bit before I take a deep breath over the weekend and try to squeeze all those 16 CT clamps into my consumer unit.
It looks me a while to get mine installed, mostly because I was daunted by the task of opening the load center. Once I started, everything went fine. Thank you for the work you guys have done. I directly uploaded the yaml with @krconv's branch, and it's working smoothly.
One thing I noticed which might help someone else to troubleshoot: If your power values see-saw between positive and negative, you have mapped the circuit to the wrong phase. I've attached a screenshot of my graph, showing before and after I fixed it to illustrate.
@flaviut's instructions are very clear on how to avoid this in the first place. My excuse is that I have a cluttered load center and somewhere along the way, I started counting wrong :)
Glad you got things working :)
I want to point out one thing: it's not unusual to get ±1W of noise in the readings, that's normal and expected. The problem your image shows is -0.25 watts (on average). Unless you have a solar system or something, that's a problem:
- you may have put the clamp on the wire backwards
- you may have selected the wrong phase (as you found & fixed)
Just stopped by to say I have converted all four of mine and they are running stable. 1 in the main panel monitoring my mains , my solar, and my dual phase circuits. 2 in the inside panel monitoring the line from the main panel, and 28 different single phase breakers 1 in my pool subpanel monitoring the pumps, controller, lighting, and UV lights.
These CT's can be finicky and I recommend calibrating them properly. The default settings were not accurate for me and depending how how close the CT is to other wires and the panel, I see a lot of variances that need to be filtered out. I chose a median filter, sliding window of 5, with a report rate of 1. I also added a lambda filter as it bounces below 0 quite a bit (a sign that you don't have it 100% calibrated) and I just want to filter those out.
The funny part is I am using emulated kasa to feed some of these circuits to my sense, just because I like their UI better.
These CT's can be finicky and I recommend calibrating them properly. The default settings were not accurate for me
About how wrong was the data for you? I'd expect up to about 5% difference because we don't set a certain calibration constant dynamically at this time. I'm curious if this was the problem.
I chose a median filter, sliding window of 5, with a report rate of 1.
Do you find this works better than a sliding average like in the example? I've ended up with this in my personal config, but if there's something better, I'd be happy to change:
- &df1
sliding_window_moving_average:
window_size: 12
send_every: 6
- &df2
lambda: 'return max(x, 0.0f);'
and
accuracy_decimals: 0
(a sign that you don't have it 100% calibrated)
I disagree. Every system has some degree of noise to it, nothing that exists is perfect. On a circuit that consumes 0W, I'd expect to see 50% of the readings below 0W, and 50% above. Are you seeing more than ±1W of noise around 0W?
For reference, here's what I see with the filter above, and I think it's reasonable:
The funny part is I am using emulated kasa to feed some of these circuits to my sense, just because I like their UI better.
That's really cool.
That's the kind of stuff I love about hacking home automation stuff. You don't need to deal with every vendor's walled garden :)
These CT's can be finicky and I recommend calibrating them properly. The default settings were not accurate for me
About how wrong was the data for you? I'd expect up to about 5% difference because we don't set a certain calibration constant dynamically at this time. I'm curious if this was the problem.
I am seeing about 5% variance now, but originally all of my numbers were trending low when compared to other manufacturer devices monitoring the same line.
When I converted these units they had been running for a while, and the calibration numbers I was seeing in the logs/NVS were higher than the defaults. I believe in both cases I was seeing different values from defaults in the source ( 0.023109 for phase A and a different number for phase B that I forgot to write down). I am assuming this was fine tuned by emporia.. I also have other CT's monitoring these same lines (The Sense, CT's in my powerwalls, my EV wall controller is also capturing them). I slowly adjusted the values until the vue started reporting similar numbers for voltage and wattage.
I chose a median filter, sliding window of 5, with a report rate of 1.
Do you find this works better than a sliding average like in the example? I've ended up with this in my personal config, but if there's something better, I'd be happy to change:
I think it comes out to a wash in the end. I am running with your filters right now (changed the windows to still report once per second) and I don't know if I would be able to tell the difference. /shrug
- &df1 sliding_window_moving_average: window_size: 12 send_every: 6 - &df2 lambda: 'return max(x, 0.0f);'and
accuracy_decimals: 0(a sign that you don't have it 100% calibrated)
I disagree. Every system has some degree of noise to it, nothing that exists is perfect. On a circuit that consumes 0W, I'd expect to see 50% of the readings below 0W, and 50% above. Are you seeing more than ±1W of noise around 0W?
In some of my lines I see it jump ±20W when at 0. When I first installed my sense, their support contacted me and let me know that if the CT was pressed against the side of my panel (rigid wiring) I would see behavior like this. I am assuming my issue is similar there for these. I am going to have to open up my panels and see if I can find a more convenient spot.
The stock settings gave me low numbers originally. I think it was showing my voltage around 10V lower than it actually was, which I think was driving all of the power numbers down. Once I had that calibrated, I am seeing something similar to you in that a 0W reading bounces around ±1W.
For reference, here's what I see with the filter above, and I think it's reasonable:
The funny part is I am using emulated kasa to feed some of these circuits to my sense, just because I like their UI better.
That's really cool.
That's the kind of stuff I love about hacking home automation stuff. You don't need to deal with every vendor's walled garden :)
I am loving this as well! Once we have arduino framework support, I am looking forward to having the device perform the kasa emulation itself. :)
One more data point about the inaccuracy of the readings - one of my circuits was showing abnormal fluctuations (+/- 30W with no load) after I bottled everything up. I went back and opened the panel and I saw that the 2.5mm jack had slightly dislodged from the port because of all the wires that are crammed in there. I reseated the connection, and everything went back to normal.
Obviously, this isn't a code issue, but for someone troubleshooting inaccurate readings, this is another troubleshooting step to consider.
One more data point about the inaccuracy of the readings - one of my circuits was showing abnormal fluctuations (+/- 30W with no load) after I bottled everything up. I went back and opened the panel and I saw that the 2.5mm jack had slightly dislodged from the port because of all the wires that are crammed in there. I reseated the connection, and everything went back to normal.
That's right, I remember now! I had this exact same issue, and actually put off putting the panel cover back on (since it would move the stuff inside around) because of it.
Thanks for bringing this up, I'll add it right to the FAQ.
Thanks for all the hard work here. I flashed stock Micropython to the esp32 and can read the CTs over i2c. Here in the UK, typically we have only a single phase. I was wondering if I can read the power factor of the circuits, and report kWh as well as kVA(h).
I was wondering if I can read the power factor of the circuits, and report kWh as well as kVA(h).
As far as I'm aware, no. There may be some configuration options for the other MCU on the board, but I've never seen them so I don't know if they exist.
Is anyone having any long term stability issues? I am using github://krconv/esphome@remove-freertos-task for a while now and I had been very happy until earlier this week when I saw that all my circuits were reading NaN.
Looking at the logs, I see this a lot:
[E][emporia_vue:055]: Failed to read from sensor due to I2C error 3
I've tried rebooting a few times and have already reflashed the firmware. I don't see how running esphome can cause this since we are only reading data from the bus, but I am curious to see if anyone else has also seen this error.
I see that same error some of the time and it usually doesn't go away without actually disconnecting the power for about 5 seconds and plugging it back in. Since mine is two phase, both circuit breakers that are providing power need to be off at the same time.
Also, you might have more luck with github://flaviut/esphome@emporia-vue. This fork appears to be more stable for me.