feature-requests
feature-requests copied to clipboard
esphome
ESP-Now integration
Describe the problem you have/What new integration you would like Integration of ESP-NOW protocol
Please describe your use case for this integration and alternatives you've tried: Just use separated esp on this protocol, but didn't know how to use esp-now with esp-home
Additional context
Similar to https://github.com/esphome/feature-requests/issues/124
New communication channels are a lot of work to add - and probably you will not see this any time soon natively supported.
Also, I don't think ESP-NOW offers many benefits for ESPHome directly - yes it (ab)uses an optimized and slimmer WiFi stack with smaller connection times etc, but with ESPHome you need a wifi connection anyway for integration with HA.
@OttoWinter
with smaller connection times
much:star: smaller connection times should be possible which should enable long running battery powered esp devices
but with ESPHome you need a wifi connection anyway for integration with HA.
Depends. One esphome node could work as a gateway for all the battery powered esphome nodes using esp-now to communicate
- Works with or without WiFi connection
- Works with or without HomeAssistant
- Instant startup time if not using wifi
- No addition hardware needed
- Easy to use pub/sub event model
- Send and receive from multiple devices
@iphong did a nice start here: https://github.com/iphong/esphome-espnow
:star2: In this video the time measured using esp-now instead of wifi was a run time of total 280ms. The same hardware build using 'classic' wifi with mqtt had a run time of around 7 seconds. This is a factor of 25 :warning:
@rradar Yeah I see ESP-NOW is an interesting protocol for cross-device communication. However, bringing this into ESPHome as an official component is a bit difficult. Of course you can create a quick component that can send/broadcast some arbitrary bytes to devices in range, but the user then is exposed very much to the inner workings of C and has to deal with bytes/etc. Also selecting which devices to send something to, or getting some sort of acknowledgement that data was received would be necessary for an official integration.
One thing that make esp-now so great is that it can work along side with home assistant. So you get the best of both worlds with no trade offs. In my opinion, espnow should not be a used as a main comunication transport. Native API and home assistant does it so well already. Since espnow doesn't rely on wifi router connection, it can be used for sensors update, or direct remote control switches. A perfect application for espnow I can think of is a 2-way or 3-way switches. Your light connect to home assistant as a usual via local wifi connection, while receiving on off command from all switches using esp-now. And your switches can get state update of that light and display it accordingly. In case home assistant somehow disconnected, your switches still work.
This is cool. I believe we need another abstraction for these
- Espnow
- LoRa
- Rs485
- can bus
- http May be, much like what web server and http components currently do
- ir, ok this one is odd
Those I believe can be target for a lightweight node to node api.
Ideally one would change the backend comm layer and node to node comm will keep working.
Api should support sending and receiving events, if the api is crafted maybe non core esphome developers (not Otto) can support those.
I'm working on a personal project that will push the esp8266 to its limit to see how good the esp-now protocol performs. The EspRC library used in my esphome integration example was part of this project.
Check it out at https://github.com/iphong/esp-visual-led
@glmnet Yeah I agree we need some abstraction to enable strutured communication over all those interfaces (I've been working on some prototypes for a while based on the current native API, but it's hard to create one that works well for all base communication layers).
One major problem with ESP-Now definitely is the missing acknowledgement support, so you can broadcast something but you never know if it arrived. which means for a real protocol we will have to invent some sort of our own TCP
One major problem with ESP-Now definitely is the missing acknowledgement support, so you can broadcast something but you never know if it arrived.
@OttoWinter actually esp-now supports encryption which should include an acknowledgement.
From https://www.programmersought.com/article/90502088859:
ESP-NOW CCMP encryption method used VSA frame. CCMP can be found inIEEE Std. 802.11-2012standard.
WiFiDevice holds a PMK (Primary Master Key) with several LMK (Local Master Key), the length of these are 16-bit Key. PWK for AES-128 encryption algorithm by LMK, call interface esp_now_set_pmk () is provided. If the PWM is not set by the user, it will use the default PWK. LMK is arranged to encrypt the VSA frame. Support for up to six LMK, VSA does not support multicast encryption frame.
But as it looks like it's not only bad documented but actually also has a hard limitation of nodes (maximum 10 or 6 depends of SoftAP/Station mode).
A very interesting approach to overcome this limitation is the EnigmaIOT protocol:
EnigmaIoT is an open source solution for wireless multi sensor systems. It has two main components, multiple nodes and one gateway.
A number of nodes with one or more sensors each one communicate in a secure way to a central gateway in a star network using EnigmaIoT protocol.
This protocol has been designed with security on mind. All node data is encrypted with a random key that changes periodically. Key is unique for each node and dynamically negotiated, so user do not have to enter any key. Indeed, all encryption and key agreement is transparent to user.
Actually the protocol only focus on the link layer which would perfectly fit with esphome which would act as the application layer:
- Application layer is not controlled by EnigmaIoT protocol but main program. User may choose whatever data format or final destination of payload. [...] The only limit is the maximum packet length that, for ESP-NOW is around 200 bytes.
- Link layer is the one that add privacy and security. It manages connection between nodes and gateway in a transparent way. It does key agreement and node registration and checks the correctness of data messages. In case of any error it automatically start a new registration process. On this layer, data packets are encrypted using calculated symmetric key.
- Physical layer currently uses connectionless ESP-NOW. But a hardware abstraction layer has been designed so it is possible to develop interfaces for any other layer 1 technology like LoRa or nRF24F01 radios.
This actually could act like the abstraction @glmnet suggested just three posts up
I am working on a ESPHome component that will use ESP-Now supporting ESP8266 and ESP32 including ESP-Now based Crypto.
Basic communication is already working great... i am just in the phase to make ist usable with yaml and a sensor integration.
May i am able to do the first release next weekend.
Emmo
After struggle with some resource problem connected to the ESP32, crypto and wifi_component, i gave up to fix this, might be there is a complete overhauling of the wifi component needed. So i release the component with the restriction that ESP32 with Wifi enabled can't communicate with ESP8266 and vice versa.
Emmo
If some one likes to check out my Wifi-Now component:
https://github.com/motwok/esphome/releases/tag/0.0.1.0
Have fun.
Also it would be great if somone does a review of the code.
Emmo
Would be great to see this functionality in ESPHome. Thanks for your work, motwok.
Even if ESP-Now is not integrated, it would be nice to have ESPHome-ESPHome communications, bypassing the HA. @iphong gives a good example using 2 and 3 way switches. I have the same situation, I have 2 Sonoff IFan-03 and I have 2 Sonoff switches to control them via HA. When the HA goes down, which it has, I loose control of the fans and the lights in the room. Even a simple request server, that can be accessed with the "HTTP Request" component would be nice, and probably much simpler to integrate. That way HA can still control the devices, but if it goes down, the devices can still function manually on their own.
Even if ESP-Now is not integrated, it would be nice to have ESPHome-ESPHome communications, bypassing the HA. This is exactly one of the reasons i created the wifi-Now module
@iphong gives a good example using 2 and 3 way switches. I have the same situation, I have 2 Sonoff IFan-03 and I have 2 Sonoff switches to control them via HA. When the HA goes down, which it has, I loose control of the fans and the lights in the room. You can do similar things with Wifi-Now Module.
Even a simple request server, that can be accessed with the "HTTP Request" component would be nice, and probably much simpler to integrate. That way HA can still control the devices, but if it goes down, the devices can still function manually on their own. An HTTP request requires at least a functioning Access Point... Wifi-Now requieres only that both modules are in range of each other.
Try it even when there is the word Alpha... it is stable enougth to be used.
Emmo
Interesting Esp-Now Flooding Mesh project: https://github.com/arttupii/EspNowFloodingMesh
Interesting Esp-Now Flooding Mesh project: https://github.com/arttupii/EspNowFloodingMesh
I made the "intresting" Esp now implementation... and i had also mesh in mind... but after wasting many hours for an Pullrequest that is completly ignored (there was a request to remove tests, no comment) like allmost all other PR with Wifi in name... i have better things to do than obeying dumb changing CI Rules and maintain PR for futhur ignorance over the next years.
The reason: the "community" is not intrested, at least the intrested people in the community do not show intrest...
mo
@motwok I tried your fork (with 'wifi' and 'wifi_now') as external components, unfortunately, I kept getting a bunch of compilation errors one after the other, seems like it does not fit with the current ESPHome internal structure anymore. Is this the case or am I doing something wrong ?!
Is this the case or am I doing something wrong ?!
I am sorry... due to an currently overvelming job i have no time to investigate this in near future.
Has this project died :( was looking forward to some low powered esp.
Nope... near future there will be an overhaul dividing it into 2 parts... one is meant as shared library and the other part is the yaml stuff
One major problem with ESP-Now definitely is the missing acknowledgement support, so you can broadcast something but you never know if it arrived.
@OttoWinter actually esp-now supports encryption which should include an acknowledgement.
From https://www.programmersought.com/article/90502088859:
ESP-NOW CCMP encryption method used VSA frame. CCMP can be found inIEEE Std. 802.11-2012standard. WiFiDevice holds a PMK (Primary Master Key) with several LMK (Local Master Key), the length of these are 16-bit Key. PWK for AES-128 encryption algorithm by LMK, call interface esp_now_set_pmk () is provided. If the PWM is not set by the user, it will use the default PWK. LMK is arranged to encrypt the VSA frame. Support for up to six LMK, VSA does not support multicast encryption frame.
But as it looks like it's not only bad documented but actually also has a hard limitation of nodes (maximum 10 or 6 depends of SoftAP/Station mode).
A very interesting approach to overcome this limitation is the EnigmaIOT protocol:
EnigmaIoT is an open source solution for wireless multi sensor systems. It has two main components, multiple nodes and one gateway. A number of nodes with one or more sensors each one communicate in a secure way to a central gateway in a star network using EnigmaIoT protocol. This protocol has been designed with security on mind. All node data is encrypted with a random key that changes periodically. Key is unique for each node and dynamically negotiated, so user do not have to enter any key. Indeed, all encryption and key agreement is transparent to user.
Actually the protocol only focus on the link layer which would perfectly fit with esphome which would act as the application layer:
- Application layer is not controlled by EnigmaIoT protocol but main program. User may choose whatever data format or final destination of payload. [...] The only limit is the maximum packet length that, for ESP-NOW is around 200 bytes.
- Link layer is the one that add privacy and security. It manages connection between nodes and gateway in a transparent way. It does key agreement and node registration and checks the correctness of data messages. In case of any error it automatically start a new registration process. On this layer, data packets are encrypted using calculated symmetric key.
- Physical layer currently uses connectionless ESP-NOW. But a hardware abstraction layer has been designed so it is possible to develop interfaces for any other layer 1 technology like LoRa or nRF24F01 radios.
This actually could act like the abstraction @glmnet suggested just three posts up
Hi,
I did not see this before.
Actually I guess it could be feasible to integrate EnigmaIOT as a EspHome component. I've just started reading about ESPHome, but I am not ready to start a development yet.
If anyone who is familiar with EspHome developint wants to try I'm open to give my help. I guess some changes would be necessary to do it but it is possible.
EnigmaIOT is quite stable now. As it solves my use case I'm not developing new features until needed.
This integration would enable EspHome to run in battery powered devices for long time, just like ZigBee but with a simple ESP32 as a gateway.
This gateway could be even modified to do a more straight integration with HomeAssistant.
Let me know any idea.
Regards
@gmag11 EnigmaIOT has a star topology. Can you expand so that it can work like Mesh Network?
This is one of my target for long term. I could use MDF sdk, but it is only compatible with ESP32. As EnigmaIOT pretends to keep compatibility with ESP8266 this will not happen soon.
You are not 100% right.
One major problem with ESP-Now definitely is the missing acknowledgement support, so you can broadcast something but you never know if it arrived.
There is a akknowledge and you get an error when not doing multicast....
But as it looks like it's not only bad documented but actually also has a hard limitation of nodes (maximum 10 or 6 depends of SoftAP/Station mode).
You can communicate with 20 registerded "Partners" and 6 different Keys (you can have 20 Stations with same key),
EnigmaIOT ...
Even with my almost 35 years experience i would not implement such security protocol by myself (ok... except for ROT13 :) ). And sorry i dont understand why people tend to reinvent the wheel? Its a mystery for me. If AES is not enough what will be enought? And please think again, having rolling keys needs some periodic communication and/or a reliable timebase, that is not an option for a low power device.
There are 2 real Problems with esp-now:
-
On receive you can not distinguish between a unencrypted broadcast and a direct send crypted packet this is the only reason for an additional encryption... and with a 32-Bit nonce and AES this should be easy enougth. So "authenticate" a packet is not possible, but needed for security applications like a low power remote for e. g. a door opener.
-
Channels... you never know on Wifi channel your "partner" (that depends in Station Mode always on the connected router) and you can receive packets when on the neightbor-channel. Switching channels to long can mess up the Wifi.
If anyone who is familiar with EspHome developint wants to try I'm open to give my help. I guess some changes would be necessary to do it but it is possible.
Unfortunately such a module is mostly one man show... to small to have multible programmers on it. If you like you can support the current development with severe testing (also developmend of tests), Documentation and code reviews. I currently divide my "wifi now" module in 2 parts, one will be a base so espnow can be used by several other modules and the other is a helper so you can do the Communication in yaml.
And for the additions it is important to overcome the channel problem (there are a few strategies but non is w/o a compromise for power consumption and timing). In a second step an oprion to use AES encryption (in addition or as alternate to the espnow encryption) will be added.
And then on top of this a proxy is needed for OTA update and HA communication (i already did a POC for nDNS)
If this is working we can do some addions for ESP-MESH! (The dream may come true)
Emmo
Hi, I add some comments about EnigmaIOT ESP-NOW implementation. I invite you to go to its repository and keel talking there
You are not 100% right.
One major problem with ESP-Now definitely is the missing acknowledgement support, so you can broadcast something but you never know if it arrived.
There is a akknowledge and you get an error when not doing multicast....
Right. unicast communication is acknowledged.
But as it looks like it's not only bad documented but actually also has a hard limitation of nodes (maximum 10 or 6 depends of SoftAP/Station mode).
You can communicate with 20 registerded "Partners" and 6 different Keys (you can have 20 Stations with same key),
With EnigmaIOT, a gateway than manage more than 100 nodes with dynamic key encryption.
EnigmaIOT ...
Even with my almost 35 years experience i would not implement such security protocol by myself (ok... except for ROT13 :) ). And sorry i dont understand why people tend to reinvent the wheel? Its a mystery for me. If AES is not enough what will be enought? And please think again, having rolling keys needs some periodic communication and/or a reliable timebase, that is not an option for a low power device.
EnigmaIOT does not try to reinvent the wheel. It uses diffie-hellman algorythm to find a common key to be used with ChaCha Poly encryption. It could use AES with some minor changes.
There are 2 real Problems with esp-now:
- On receive you can not distinguish between a unencrypted broadcast and a direct send crypted packet this is the only reason for an additional encryption... and with a 32-Bit nonce and AES this should be easy enougth. So "authenticate" a packet is not possible, but needed for security applications like a low power remote for e. g. a door opener.
EnigmaIOT solves this with a few bytes of metadata. ChaCha Poly encryption does authentication. That was one of the reasons I chose it.
- Channels... you never know on Wifi channel your "partner" (that depends in Station Mode always on the connected router) and you can receive packets when on the neightbor-channel. Switching channels to long can mess up the Wifi.
You can broadcast an AP on gateway to mark the channel. Nodes find that AP then try to authenticate to it. This avoids gateway impersonation. After that they disconnect from AP and continue taking through ESP-NOW. In case of a channel change in gateway, nodes try to find it again after a timeout. Authenticated encryption allows to ignore out-of-network messages.
If anyone who is familiar with EspHome developint wants to try I'm open to give my help. I guess some changes would be necessary to do it but it is possible.
Unfortunately such a module is mostly one man show... to small to have multible programmers on it. If you like you can support the current development with severe testing (also developmend of tests), Documentation and code reviews. I currently divide my "wifi now" module in 2 parts, one will be a base so espnow can be used by several other modules and the other is a helper so you can do the Communication in yaml.
And for the additions it is important to overcome the channel problem (there are a few strategies but non is w/o a compromise for power consumption and timing). In a second step an oprion to use AES encryption (in addition or as alternate to the espnow encryption) will be added.
And then on top of this a proxy is needed for OTA update and HA communication (i already did a POC for nDNS)
You can tigger a node to connect to wifi to get an OTA update. This is not developed right now. Currently a python script allows to send OTA update through MQTT to gateway and then through esp-now to nodes.
If this is working we can do some addions for ESP-MESH! (The dream may come true) Sure, this would be super
Emmo
If ESP-Now LR does really improve the WiFi capabilities to 480 meters and adds mesh functionality it would be a nice alternative to Zigbee.
Would be super interesting to have ESP-Now LR to tunnel information to a dedicated ESP32 which connects to the HA instance like a hub.
In theory this only needs a dedicated port on the hub for each "endpoint" in the ESP-Now network. On the "endpoint" node we convert it back to TCP/IP packages and receive it via the normal tcp/ip stack.
This should in theory allow all regular features except mDNS discovery.
@RubenKelevra wrote
adds mesh functionality
I've looked a bit deeper into EPS-Now, while it can do mesh as in "everyone connects to everyone" it can't route packages like Zigbee routers do.
So we could do multiple-start topology, where every device with a Network-Uplink is working as a hub while the devices will be available via each of those hubs. In this case HA would just try each of those forwarded ports until it reaches the node.
Say we start with 10000 as port for the first node, on each hub the port would end up on the same node in the ESP-Now network. So HA would get a list of hubs and just try to reach the node via each of those IPs with the port 10000.
This allows for redundancies and mobility, while the connection might dropout for a short duration, it will be picked up again as long as the node is within the range of an ESP-Now "hub".
The other solution would be to do routing via a simple implementation of Batman-adv. That's a protocol to route L2 packages inside a L2 network.
The idea is to send out a broadcast-package from time to time to determine the "best" next hop to each hub. This ways an ESP-Now "endpoint" would just need to store the best reachable mac address towards the hub and send the packages to this node.
Since we don't need the full length of a mac address to address a hub we might loose just 1-2 bytes for this "routing" as each package need to contain the selected hub (by enum number).
Another option might be, to implement a minimal subset of 802.11s with a smaller channel width (if the hardware is capable of that). Using a 5 MHz channel, you could (at least in Europe) "squeeze" the ESP32/8266 network above channel 11, as we have up to channel 13 available here.
The default HWMP protocol for routing isn't possible to use tho, as all nodes need to keep track of the whole network.
But using ideas from Batman-adv (like pointed out earlier) we could get away with just storing just "next hop" informations to the "hubs".
The main advantage compared to ESP-Now would be, that 5 MHz channels are also supported by OpenWRT (depending on the hardware capabilites). So the network could just be routed in the local router or the (in most setups) unused Wifi card in the Rasberry Pi could be used as hub for this network.
This would on one hand eliminate all broadcast/multicast traffic to be received and processed by the ESPHome nodes and on the other hand allow for low power hop-to-hop communication of the ESPHome nodes in a mesh network, extending the range of individual nodes.
While this might not reach the extremly large distances of the ESP-Now, this on the other hand does not need any tunneling or something like this - as the package size is not restricted.
TL;DR
So this would reduce the power consumption, the processing overhead for wifi, allow for long range communications via hops, make the connections more stable and also reduce the latency - as we don't have to wait for free airtime as much as with a 20 MHz channel in the crowded 1, 6, 11 channels.
@RubenKelevra wrote ...
Here is the deal: Let us finish esp-now... currently i am recovering slow from an temporary blindness... so it will take a bit of time.
If this is done you might use the base module in your own to realize this stuff you describe (may be you make something meshy) but be aware that there are already a mesh implementation for ESP32 that works reliable.
So in mesh (and also esp-now) the biggest Problem is that almost every thing in ESP Home relies on TCP/IP for example try the MQTT component having communicating to a serial partner or pipe instead of TCP/IP connection (i tryed already... upsi, the library used includes the TCP/IP stuff... yust rewriting is needed)... When thinking on most components, thats a sysipus task... So my approach is to have something simple and easy to implement: a simple port-proxy that has the cababiity to make client connections over a "hub"-module and accepts connections on the "hub"-Module including announcments via mDNS, may be its possible to avoid manual configuration for the incoming connections.
We will see in the next few month.
Emmo
@RubenKelevra wrote ...
What you propose is not legal in most countrys, because you modify Layer 1 and loos the certification for your ESP Module... running those modifications is only legal for HAM Operators. And real low power with Wifi-Routing , what do you dream at night, its like try light a match under water, sorry.
And Whhhhyyyy at all??? Again, an advanced selforganizing mesh is already implemented, no need to f..k around with illegal WIFI-Protocol tricks, its just there, it works and you can use it.
The only thing to do for mesh in ESPHome is the communiction from a mesh-node to TCP/IP. Thats all you need. (Having a proxy solves this problem)
Emmo