airbnk_mqtt
airbnk_mqtt copied to clipboard
rospogrigio
ESPHome
Hi @rospogrigio ! Long time :)
I recently developed custom BLE presence for HA, based on ESP32, and meanwhile realized, how to get it working as custom ESPHome component. Do you think it's worth to move my custom gateway firmware to similar component? It would require 1 file and 6 lines of code to add it.
I actually think, that it would be useful for new users. Tasmota is good enough, but ESPHome is widely used with HA. E.g. someone could install this to SONOFF NS Panel. :)
Sounds interesting but I'm not sure I am understanding this fully, especially because I don't know much about ESPHome, can you add some more details to explain, and pros and cons of the thing? Thank you!
Well, ESPHome allows to have ESP configuration and updates at one place, and it's really convenient to create firmware, flash it, track devices, read logs, OTA update etc.
Custom components to ESPHome are C++ .h files, that basically can use esphome.h library, and be included into ESPHome configuration.
For example, check corresponding section in my BLE presence repo here
Well, looks promising, let's try it! Will we need to change anything in the integration?
Hi @formatBCE , @rospogrigio
It possible to add also fingerprint and keypad example for sensor battery ?
thx for all . BR Emil
It possible to add also fingerprint and keypad example for sensor battery ?
Sorry I don't understand your question, can you explain more clearly? Thank you
for this lock they also have pinpad and keypad that working with bluetooth also and connecting with the same app - WeHere
it will be great to add them inside to monitor their battery status and who loged in .
thx
Well then I believe it's not possible (I own the fingerprint command too), I believe that the ESP32 can connect to one device at a time but @formatBCE might correct me. In any case, I believe that they communicate with a protocol different form the lock's one... I think all the effort is not worth it, only for checking the battery level.
Ok , thx for quick response.
On Sun, 4 Sept 2022, 15:43 rospogrigio, @.***> wrote:
Well then I believe it's not possible (I own the fingerprint command too), I believe that the ESP32 can connect to one device at a time but @formatBCE https://github.com/formatBCE might correct me. In any case, I believe that they communicate with a protocol different form the lock's one... I think all the effort is not worth it, only for checking the battery level.
— Reply to this email directly, view it on GitHub https://github.com/rospogrigio/airbnk_mqtt/issues/24#issuecomment-1236332423, or unsubscribe https://github.com/notifications/unsubscribe-auth/APONFSXIIWL7WKRMJXDX423V4SKN5ANCNFSM6AAAAAAQAANSSQ . You are receiving this because you commented.Message ID: @.***>
@rospogrigio hello! Working for 15 minutes a week (kids...), I think I have somewhat working custom gateway component, which can be installed with ESPHome.
In parallel (having convenient logging, provided by ESPHome), I discovered problem with gateway itself. It's not so visible on custom firmware, but on ESPHome I clearly see it.
Case is, that on ESP32 antenna is used by WiFi and BLE simultaneously. While simply scanning, it's managed by scan window and interval params (basically, BLE is using antenna for 64 ms, and then releases it for 64ms for WiFi).
But with commands it's different story: for sending command to lock, gateway should establish connection to device, service and characteristic to write data. Then gateway should get back write status to report back to MQTT. This takes time, and WiFi does sometimes disconnect (especially if first connect try was unsuccessful, and we're retrying). Then after writing data, gateway tries to send status to MQTT. Which is impossible actually because WiFi is reconnecting. So this answer might be lost.
I temporarily patched it by adding MQTT connection check on status sending, but main problem persists: ESP32 isn't created for simultaneous extensive usage of BT and WiFi.
My main topic here is following:
- could whole logic be moved to C++, so I could use it in ESP?
- if not, maybe, we could move everything payload-related? I saw tools/ folder, and looks like it has everything, except of initial tokens, that could be provided by integration - and whole BLE related stuff would be on ESP32. This way we'd remove strong relying on response for integration - it would just describe info from MQTT instead of waiting in "operating" mode.
Think about it. Worth it?
OK that looks interesting and explains some realiability issues. I don't understand the 2 final questions though:
- isn't it already in C++? Your firmware, I mean
- what do you mean by payload-related? I don't understand the whole sentence here actually, can you explain it a little bit more? Maybe you want to move the generation of the commands within the ESP, and the Integration should provide only the needed keys at the first connection, is this what you mean?
Also, how long does the handshakes take, in terms of ms? And how about the 64ms slots, are they defined by the firmware?
Sorry for being unclear here. Yes, I'd like to check possibility to move payload generation to ESP. It would help to loose coupling between HA and ESP, easing status tracking and handling errors (basically, HA integration would be responsible only on setup/interface part).
That window of 64 ms (as well as interval of 128 ms) is fully customizable. But don't focus on this, as that part (scan) works perfectly now. What is giving problems, is command execution (it's a bit different from passive scan). However, yesterday I drastically optimized that part (and also published custom component for ESPHome setup). Didn't do it for main firmware yet (there's different code base because of ESPHome API), but on ESPHome gateway I did like 20 consecutive lock/unlock operations, and they all passed without error. So I consider this mitigated (though not fixed). Handshake/connection is pretty fast, unless there's intermittent error connecting, when we're retrying. Also additional time is needed to read from that status characteristic after writing (it almost always takes 3-4 tries with 700 ms in between) because of that ...00000 value, you remember.
Do we actually need it btw? If ESP just sends success, won't it be enough? It could help drastically, if we skip that. As I see, you're using only adv data for codes generation - but you will receive that in next scan.
Yeah I questioned myself many times about this. One issue could be if the meanwhile it receives a command via the app, but we probably can consider this event as very unlikely. I seem to remember that actually the lock would not accept further commands until that status characteristic is updated but I could be very wrong in this remembrance. The only thing we actually need from that status is the lockEvents as you know, but it could just be incremented on successful commands... I suggest you try removing it and see if it works or you get any issues.
I would have to remove it from integration too, right? Because there's logic, that will fail operation result, if there's no correct response from characteristic.
What about lockEvents - isn't it gets updated on each advertisement message? I see it in code.
I will try to remove this logic from gateway and integration and check result.
Ok, i removed need of status payload from integration, and disabled reading from characteristic in gateway. From my feeling, it works WAY better. Status of lock is updated with next advertisement, and consecutive commands are working good. There's a bit more work on synchronizing state in HA, but overall it's really good. Now it's lagging only, if lock is lagging itself (reconnections).
P.S. Will use it like this for some time to ensure it's all good, and maybe post PR.
@rospogrigio today I realized, that payload shouldn't be generated on ESP, because for this sensitive fields, used for generation, will have to be transferred over MQTT from HA integration - and this should be considered insecure channel. So I guess we must leave it like this.
Well, I understand your point but we can think about this. I have always felt a bit of paranoia around this because I believe the risk is extremely low, because in this case to gain access to your house an attacker should:
- be physically in front of your door
- know that you own a remote lock, which one it is, and that you are using a custom gateway too
- have broken into your wifi LAN (and this should not happen if you use WPA2 and a strong enough password)
- know the integration you are using, how it works and how to intercept the codes ...so I believe that the event of a successful hack is extremely unlikely.
On top of that, I believe that the keys should be sent to the gateway only at its startup, so he should intercept the codes right at that moment because they would not be exchanged in any other moment. If you still feel this to be a bit too risky, maybe it's possible to use one of these methods to send the keys to the lock in a secure way:
- make the user store them within the web configuration of the gateway, and have the keys stored in the configuration write-only. To increase this method's security however the configuration should run over HTTPS and I don't know if it's easy/feasible
- maybe it's possible to generate a pair of public/private RSA keys on the gateway and encrypt the keys before sending them to the gateway?
Just a couple of ideas that came into my mind, let me know what you think... Great job in the meantime! 👍
Thanks for suggestions! Today i found that BLE library for ESP32 allows continuous scanning instead of restarting all the time we received adv from lock. It allows to send each and every adv message to HA, which results on lightning-fast status update. Testing on my setup and have no problems yet.
About my security concerns - i'm assuming that some users can use insecure MQTT (e.g. public MQTT servers) - and this can lead to exposure. After that, possible intruder just have to bring device with gateway firmware to the door, that's it.
On the other hand, with current setup the only thing that can be catched on MQTT is one-time command, which won't help breaking in.
I understand, that actual burglar would just kick the door or window out, but don't want to add another way to get long-term backdoor to the lock (which actually could be untraceable).
On adding config to gateway - yes, it can be done, e.g., on ESPHome config step, from secrets file. However, i don't see good way to get that values without scraping debug logs.
Encryption would work, i guess, but it would add more complication to message handling, and slow down operation drastically (ESP chip is not for encryption all-in-all) :)
I was already thinking on getting everything, including API calls, to the gateway. This way everything would be bound to ESP exclusively. Don't know if it's worth trying though. It's a heck of work :)
I'd like to know, if we're ready to get rid of status characteristic. I removed related code from my installation, and it works stable and reliable.
Just to share my excitement: I just successfully merged two of my ESP BLE projects into one ESPHome-based component, and installed to one ESP32. Also i did install this to Sonoff NSPanel, which is conveniently located right beside my front door, so it will be both picking up my Tile beacons on approach, and rule my lock :)
I'd like to know, if we're ready to get rid of status characteristic. I removed related code from my installation, and it works stable and reliable.
Well, try it yourself and then you can submit a PR. I'm not sure I followed everything you've been trying but if you have something to share I'd be happy to try it on my setup and provide feedback. Let me know! 👍
Hi @formatBCE , I'd need your help... I had moved to the ESPHome-based component as you suggested, and everything was working quite fine. Now, I have upgraded my RPi 4 to Debian bullseye (which was a huge struggle, but I finally managed to upgrade everything), and everything is working.... except for the airbnk. The ESPHome logs keep saying:
[16:04:02][D][airbnk_mqtt:123]: Sending adv
[16:04:04][I][mqtt:176]: Connecting to MQTT...
[16:04:04][W][mqtt:264]: MQTT Disconnected: TCP disconnected.
[16:04:04][D][airbnk_mqtt:105]: BLE scan heartbeat
[16:04:05][D][airbnk_mqtt:123]: Sending adv
[16:04:06][D][airbnk_mqtt:123]: Sending adv
[16:04:09][D][airbnk_mqtt:123]: Sending adv
[16:04:09][I][mqtt:176]: Connecting to MQTT...
[16:04:09][W][mqtt:264]: MQTT Disconnected: TCP disconnected.
[16:04:10][D][airbnk_mqtt:105]: BLE scan heartbeat
[16:04:11][D][airbnk_mqtt:123]: Sending adv
[16:04:12][D][airbnk_mqtt:123]: Sending adv
The mosquitto instead logs:
1667744264: mosquitto version 2.0.11 running
1667744462: New connection from ::1:44493 on port 1883.
1667744462: New client connected from ::1:44493 as 1F9uyNVRPBN6z1KR6doZqb (p2, c1, k60, u'airbnk_mqtt').
1667744923: Client 1F9uyNVRPBN6z1KR6doZqb closed its connection.
1667745021: New connection from ::1:55317 on port 1883.
1667745021: New client connected from ::1:55317 as 5YpDtxFU3zLDOiiGmk74qH (p2, c1, k60, u'airbnk_mqtt').
1667745431: Client 5YpDtxFU3zLDOiiGmk74qH closed its connection.
1667745591: New connection from ::1:48367 on port 1883.
1667745591: New client connected from ::1:48367 as 3nE0vFsPmNrAvGi3SblAmO (p2, c1, k60, u'airbnk_mqtt').
1667745787: mosquitto version 2.0.11 terminating
1667745787: Saving in-memory database to /var/lib/mosquitto//mosquitto.db.
1667745877: mosquitto version 2.0.11 starting
1667745877: Config loaded from /etc/mosquitto/mosquitto.conf.
Can you help me understand what's going on and how I can solve these issues? Thank you in advance...
Hi @rospogrigio !
Huh, that's tricky one. Without context, I'd say that ESP is dropping TCP connection because of conflict with BLE, and would ask to try different ESP... But, obviously, you're using same chip, as with old rPi OS, right?
Did you maybe update ESPHome to latest version, and upgraded ESP with it?
My gateway is still on ESPHome 2022.10.0, although there's update to 2022.10.2 available, and they made some BLE stack update there. Probably, it could conflict somehow...
For now it looks to me like gateway problem, not the rPi.
If you eager to play with it a bit, I suggest to:
- try custom firmware back - does it work?
- try downgrade ESPHOME to 2022.10.0
Please let me know, if something of that works. :) Meanwhile, I won't be upgrading to 2022.10.2...
Mmm I believe I did update that. How can I check the installed version? And how can I rollback?
Looks like there's no good way to do it - only with rollback from backup... But now we know at least, that this is possible trouble.
I'll try to update to 2022.10.2, and we'll see, of it's working for me.
So I updated gateway (rather flashed another chip) with 2022.10.2. It works. I had one message about TCP disconnected, but it recovered right away, and not seen after several reboots.
Gosh! So what can I try to debug it? Any suggestions? Edit: can you post here what your ESPHome log is displaying? I can't remember what I'm supposed to see there, so I can't understand if it's not talking to the lock or if it's not sending data to the broker... thank you
As I suggested, could you put custom firmware to ESP, and try? The logic there is same, so we could test, if it's MQTT dropping connection, or ESPHome does bad thing.
OK, flashed... no luck. Here are the logs I get via serial on the ESP32:
[00:56:20]Connecting to MQTT
[00:56:20]Disconnected from MQTT.
[00:56:20]Starting MQTT reconnect timer
[00:56:20]restarted
[00:56:21]Scanning done.Error sending telemetry
[00:56:21]Scanning...
[00:56:21]Cannot report: mqtt disconnected
[00:56:21]Time remaining: 895
[00:56:21]Cannot report: mqtt disconnected
[00:56:21]Time remaining: 644
[00:56:22]Connecting to MQTT
[00:56:22]Disconnected from MQTT.
[00:56:22]Too many retries. Restarting.
[00:56:22][WiFi-event] event: 3
[00:56:22]STA Stop
[00:56:22]ets Jun 8 2016 00:22:57
[00:56:22]
[00:56:22]rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
[00:56:22]configsip: 0, SPIWP:0xee
[00:56:22]clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
[00:56:22]mode:DIO, clock div:2
[00:56:22]load:0x3fff0030,len:1184
[00:56:22]load:0x40078000,len:12784
[00:56:22]load:0x40080400,len:3032
[00:56:22]entry 0x400805e4
[00:56:22]Connecting to WiFi...
[00:56:22][WiFi-event] event: 0
[00:56:22]Wifi Ready
[00:56:22][WiFi-event] event: 2
[00:56:22]STA Start
[00:56:22]WiFi Timer is inactive; resetting
[00:56:23]restarted
[00:56:23]restarted
[00:56:25][WiFi-event] event: 4
[00:56:25]Connecting to WiFi...
[00:56:25][WiFi-event] event: 5
[00:56:25]WiFi lost connection, resetting timer
[00:56:25]restarted
[00:56:27][WiFi-event] event: 4
[00:56:27][WiFi-event] event: 7
[00:56:27]IP address: 192.168.1.10
[00:56:27]Hostname: airbnk_topic (I believe this is a bug, this is actually the name of the topic)
[00:56:27]Connecting to MQTT
[00:56:27]Scanning...
[00:56:27]Stopping wifi reconnect timer
[00:56:27]Cannot report: mqtt disconnected
[00:56:27]Starting MQTT reconnect timer
[00:56:27]restarted
[00:56:27]Cannot report: mqtt disconnected
[00:56:27]Starting MQTT reconnect timer
[00:56:27]restarted
[00:56:27]Disconnected from MQTT.
[...]
[00:56:36]Cannot report: mqtt disconnected
[00:56:36]Time remaining: 1516
[00:56:37]Scanning done.Error sending telemetry
[00:56:37]Scanning...
[00:56:37]Cannot report: mqtt disconnected
[00:56:37]Time remaining: 1052
[00:56:37]Cannot report: mqtt disconnected
[00:56:37]Time remaining: 699
[00:56:38]Connecting to MQTT
[00:56:38]Disconnected from MQTT.
[00:56:38]Starting MQTT reconnect timer
[00:56:38]restarted
[00:56:40]Connecting to MQTT
[00:56:40]Disconnected from MQTT.
[00:56:40]Starting MQTT reconnect timer
[00:56:40]restarted
[00:56:40]Cannot report: mqtt disconnected
[00:56:40]Time remaining: 1656
What now?
Ok, it's not an ESPHome. It's most probably mqtt, or TCP itself. Which makes sense.
Is MQTT reachable from other addresses? Try to check, what's wrong with it - looks like it doesn't allow connection...
It might be TCP connection trouble, or something with MQTT settings - unfortunately, can't tell precisely...