ATC_MiThermometer
ATC_MiThermometer copied to clipboard
pvvx
New esphome component for pvvx firmware + "Custom" advertising
Available here : https://github.com/airy10/pvvx_mithermometer
Mainly a copy of the esphome atc_mithermometer component with updated decoding code
More relevant is support for Xiaomi Mijia encoding and fix drops when receiving ads in ESP32. I have not found any examples with AES CCM decoding for ESP32.
Ads format from the original firmware is already supported by the bundled xiaomi_lywsd03mmc components. Or you mean custom format + encryption ?
ESPHome does not work on Android and Windows. Requires specific knowledge, additional costs for unnecessary components and assembly of the system from ordinary users. In ESPHome It is impossible to solve the simple problem of receiving and parsing advertising packages for the entire BLE group, which has a couple of formats directly on ESP32.
Well, ESPHome is working great for me as a cheap Wifi bridge for my LYWSD03MMC devices and I just thought this component might be useful for other people too...
The component can be useful when fixing ad-packet reception skipping in ESP32 drivers. Otherwise it is not a working solution.
Well, ESPHome is working great for me as a cheap Wifi bridge for my LYWSD03MMC devices and I just thought this component might be useful for other people too...
I can see the benefit when using Home Assistant or other Home Automation. In my case, I need additional ways to listen for advertising packages (home automation server bluetooth does not reach all corners of house) and relay them back to main server; so in my setup (Xiaomi LYWSD03MMC->ESP32 Tasmota->Node Red->Home Automation Server). I have not had any joy with ESPHome and my ESP32s, so curious if this would work
ESPHome, ESP32-wrover-b, 3 sensors are registered.
Half an hour charts. Each graph must contain 180 points (!):
Log Debug says that only 0.5 AD packets per second were received from all surrounding BLE devices.
The Sniffer says that the loss does not exceed 5% of the packets.
I need additional ways to listen for advertising packages (home automation server bluetooth does not reach all corners of house) and relay them back to main server;
Reception of advertisements on JDY-10 (TLSR8266) or CC2540 USB Dongle. https://github.com/pvvx/UBIA/tree/master/Sniffer/OpenWRT_SnifferAd Reaches several hundred ad packages per second... https://www.youtube.com/watch?v=MqFcY5Hovpw
Even if packets are lost, even receiving new values for temp and humidity every couple of minutes is good enough for me for that kind of sensors. I'm using it with Home Assistant for sensors that are too far from my Raspberry Pi. I can understand that it might not fit some people needs, but it's very much good enough for my needs, especially for the price and easy of use.
Le 12 avr. 2021 à 17:33, Victor @.***> a écrit :
I need additional ways to listen for advertising packages (home automation server bluetooth does not reach all corners of house) and relay them back to main server;
Reception of advertisements on JDY-10 (TLSR8266) or CC2540 USB Dongle. https://github.com/pvvx/UBIA/tree/master/Sniffer/OpenWRT_SnifferAd Reaches several hundred ad packages per second... https://www.youtube.com/watch?v=MqFcY5Hovpw
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.
Even if packets are lost, even receiving new values for temp and humidity every couple of minutes is good enough for me for that kind of sensors.
Smart Home users are not interested in sensor readings. They are interested in the execution of scripts, which requires a certain response of timely triggering according to data from the sensors. And also the reliability of the system, which is not provided in the HA.
According to the scheme of building a smart home system with ESPHome and BLE, the most unstable component today is ESP32. Due to the instability of ESP32 reception and operation, difficulties with the installation for ordinary people of ESPHome, I cannot advise users to use ESPHome. The situation is similar with "HA" - there is no support for standard devices for placing the "HA" system on the existing equipment of users. Typical equipment is old cheap smartphones, computers, routers and other household junk.
Smart Home users are not interested in sensor readings. They are interested in the execution of scripts, which requires a certain response of timely triggering according to data from the sensors.
Indeed, in the case of my switches and automations. My heating is a different story (updates every minute).
According to the scheme of building a smart home system with ESPHome and BLE, the most unstable component today is ESP32.
Guess I've been lucky, as these have been stable without a reboot/restart for over 4 months now.
Typical equipment is old cheap smartphones, computers, routers and other household junk.
..or in my case a few thousand Euros of equipment; but understand many HA systems are hacked together bits and pieces that require never ending maintenance to keep running.
Guess I've been lucky, as these have been stable without a reboot/restart for over 4 months now.
This is another advertisement without confirmation.
Xiaomi LYWSD03MMC with custom firmware is able to switch heating without assistance.
Don't know if anyone cares, but my results with ESPhome on ESP32 have been outstanding with my three reflashed MiThermo sensors. This one is even locked up inside my refrigerator, and I am very interested in the charts recorded by HA to see how my fridge performs....
PS. My HA system IS hacked together with bit & pieces, and once I ironed out a few bugs (like SD card failures) and run a SSD drive with the RPi (instead of a big SD card) the reliability has also been outstanding.
Guess I've been lucky, as these have been stable without a reboot/restart for over 4 months now.
This is another advertisement without confirmation.
Xiaomi LYWSD03MMC with custom firmware is able to switch heating without assistance.
Here is the confirmation. Data at 5 minute frequency written to Domoticz. My heating system is not an On/Off, but a Modulating boiler that is adjusted based on heating requirements for my underfloor heating on the ground floor, wall mounted radiators on other floors and control valves for these, combined with the heat loss/retention properties of the different rooms. So, unfortunately a LYWSD03MMC doesn't meet my needs as the thermostat. I'm certainly not 'advertising ESP32s', as mentioned perhaps I've been lucky so far.
Graphs in the thermometer itself are recorded in more detail. And we are talking about the speed of receiving information and without dropout. Actual for PID and more accurate adjustments.
Show on your schedule actually accepted glasses that the thermometer transmits every 10 seconds. Not a diagram that has interpolation to display.
ESPHome with ESP32 forces you to duplicate ad packets and transfer them as often as possible, which greatly affects the battery consumption of the sensor. With an advertisement interval of 2.5 seconds with a sensor polling every 10 seconds, 12 packets are transmitted. Any other BLE receiver in noisy radio air, it loses no more than 8% of packets, then ESP32 in ESPHome loses more than 90%. With an increase in the number of devices, this is already critical.
Comparison of the modified hcitool says that the ESP32 program needs to be fixed.
I've also observed issues with using ESPHome for receiving BLE advertisements. I've spent a little bit of time trying to fine tune the scan_parameters, but the performance is still pretty abysmal.
Fundamentally I think the problem is that an ESP32 can't be listening for BLE advertisements while simultaneously sending/responding to network requests over WiFi. I suppose you could set one up to use ethernet instead, but it seems like a rather roundabout solution.
Here's a graph of the number of advertisements received over a 24 hours period of three of my BLE sensors grouped by periods of 5 minutes. While I certainly don't expect this to be a perfectly flat line, less than one advertisement being captures for a given sensor over a period of five minutes just doesn't work for the use cases I have in mind.
@
Fundamentally I think the problem is that an ESP32 can't be listening for BLE advertisements while simultaneously sending/responding to network requests over WiFi. I suppose you could set one up to use ethernet instead, but it seems like a rather roundabout solution.
I guess one way to prove where the issue is in the ESP32 system is combine an ESP32 with a ESP8266 (or second ESP32), and send the received data from the ESP32 listener (without any WiFi functions) to the ESP8266 via serial for the ESP8266 to send to HA (or wherever the target is). Could use MQTT instead of ESPhome service as well... Or is there a better HW just as cheap as an ESP32 we could use ?
I guess one way to prove where the issue is in the ESP32 system is combine an ESP32 with a ESP8266 (or second ESP32), and send the received data from the ESP32 listener (without any WiFi functions) to the ESP8266 via serial for the ESP8266 to send to HA (or wherever the target is). Could use MQTT instead of ESPhome service as well...
Questions about missing advertise packages for ESP32 are available on many Internet resources. The selection of scan intervals does not help.
For comparison, I tried many options:
- BLE Sniffers / TLSR8266
- Modified hcitool
- RTL8722DM
- USB-BT adapters also differ in the number of AD packets received per unit of time: https://www.youtube.com/watch?v=3getuFLcMko
MQTT on weak platforms limits performance and does not have time to service a large network of sensors. mqtt-performance-tests
Or is there a better HW just as cheap as an ESP32 we could use ?
Xiaomi LYWSD03MMC. TLSR8251 has USB2.0 FS. USB pins are available on the board... JDY-10 (TLSR8266) = $1 TB-03F, TB-04 (TLSR8253) BLE, ZigBee, ... BW16 (RTL8720DN) BLE + WiFi 5GHz, USB2.0 HS ...
Note that while the ESP32 will never be the device to use if you need high availability for BT+Wifi, you can improve BLE adv reception by enlarging the scanning window, which is very small by default. I'm getting about 35% of them with these parameters in a quite noisy environment - which is enough for what I'm using it for - you'll decide if it is good enough for you :)
esp32_ble_tracker:
scan_parameters:
active: false
interval: 211ms
window: 120ms
Default is 350ms/30ms (using it my LYWSD03MMC using 10s advertising interval/ and x10 measure interval - which is much less that the default setting but I don't need more points) I wouldn't use that for life-saving features, but it's definitively good enough for me for remote checking temperatures which aren't changing quickly).
What prevents the RF receiver in ESP32 from working permanently? The CPU does not take part in the reception. Usually, a packet received by the radio path causes an interrupt and is processed by DMA. But in reality, it turns out that the proprietary ESP32 drivers cannot cope with the 1 Mbit / s stream. :) In the Espressif documentation, the reception level in dB is higher than that of other cheap ICs, but in fact the opposite is true. Perhaps Espressif has no specialists or there are errors in the microcircuit...
I compared the number of dropped ESP32 packets without Wi-Fi enabled with the output to the UART. Changes to scan parameters have very little effect on packet rejection rate. At the same time, other chips with weaker processors and characteristics of the radio path have a lower percentage of drops. Perhaps this is due to a malfunction of the automatic reception level control system in the ESP32. But it is more likely that these are algorithms applied in the RF driver itself.
It was noticed that the processing time of data from ad packages also affects the number of lost packages. It looks like the drivers are not buffered and not designed to work with two cores ...
I appreciate the insights @pvvx -- honestly this isn't something I've had much time to research thoroughly.
Are there any "plug and play" solutions that could be either used with MQTT or easily adapted to use with Home Assistant? Not that I don't like tinkering with these things, but my time is at a premium right now. Part of the appeal of ESPHome was the fact that the time it took to set it up was trivial.
@kelchm you can use the custom component below to integrate your devices with home assistant. It works great.
https://github.com/custom-components/ble_monitor
This is from my custom setup, I use my own sketch on ESP32 for BLE -> MQTT deal, influxdb for database and grafana for presentation. I also log via uart each packet, I get one every few seconds. What bothers me more, is the fact that your firmware drains the battery faster (default settings) than the 'original' atc1441 as can be seen here:
I confirmed it using my multimeter.
What prevents the RF receiver in ESP32 from working permanently? The CPU does not take part in the reception. Usually, a packet received by the radio path causes an interrupt and is processed by DMA.
From what I understand, WiFi and BLE are multiplexed, thus you can't be in 'BLE rx' mode all the time.
One important note, why is this important: ESPhome + temp sensor (as those ble ) is perfect solution for smart automation, especialy heating. ESPhome has native support of checking availabilty of HomeAssistant ( whole automations can be run in super "smart" way, HomeAssistant+NodeRed+whatever you want) but in case of power outage, server failure etc, ESPhome can automaticly "switch" to safe mode and act as regular thermostat (maybe little bit smarter with some simple automations written directly in esp). Im using this "safe" scenario even for wall switches.
Your graph should have 1080 points in 3 hours. At default settings, the data changes every 10 seconds, but this is not on your graph. 80% of packages are missing. 3 hours x 60 minutes 60 seconds in 10 second increments creates 1080 points. And on the presented chart there are only about 160 points in 3 hours. :)
What bothers me more, is the fact that your firmware drains the battery faster (default settings) than the 'original' atc1441 as can be seen here:
Is the battery charging in the center of the graph? :)
What prevents the RF receiver in ESP32 from working permanently? The CPU does not take part in the reception. Usually, a packet received by the radio path causes an interrupt and is processed by DMA.
From what I understand, WiFi and BLE are multiplexed, thus you can't be in 'BLE rx' mode all the time.
Testing was carried out with the pin in the UART. WiFi is off.
MHO-C401 records from Flash Memo. Step 10 minutes.
The battery (СR2032) voltage always repeats the temperature schedule.
Your graph should have 1080 points in 3 hours. At default settings, the data changes every 10 seconds, but this is not on your graph. 80% of packages are missing. 3 hours x 60 minutes 60 seconds in 10 second increments creates 1080 points. And on the presented chart there are only about 160 points in 3 hours. :)
What bothers me more, is the fact that your firmware drains the battery faster (default settings) than the 'original' atc1441 as can be seen here:
Is the battery charging in the center of the graph? :)
I did a query on my db, around 400 points every hour. Disabling logging to flash made all the difference in regard to power consumption. Thanks for your hard work man, all this makes the device very attractive.