ElegantOTA
ElegantOTA copied to clipboard
ayushsharma82
[Must Read] V3 Project Plan & Features
Hello everyone!
It has been 3 years since I created this library and it has received unprecedented love from the open-source community which has been the driving force to keep enhancing this library.
Today marks as a stepping stone for V3 of ElegantOTA which will further reduce the size of webpage, add exciting new features, potential bug fixes with new optimized source code and most importantly support for new microcontrollers.
To make this possible, I'll need your help in figuring out what's best and what you guys need in ElegantOTA. So comment down any feature you feel is missing in ElegantOTA and I'll add it in my project plan.
Supporting New Platforms: For past 3 years ElegantOTA has been stuck with ESP32 and ESP8266 but now I wish to increase the coverage of ElegantOTA so that more people can make use of it to update their microcontroller conveniently. If you know or work with any microcontroller ( other than ESP32 or ESP8266 ) that has OTA capability please write it down in the comments below. I'll look into it.
The microcontroller doesn't necessarily needs to have in-built WiFi, it can also be a NCP ( Network Co-Processor ) architecture like Nano 33 IoT.
Hey, @ayushsharma82, How about the compressed updates support? For esp8266 it's pretty easy out of the box since it's supported by eboot code, just need to add file format detection. For esp32 you need some additional code to decompress an upload on the fly. Not so long ago I've made a small lib implementing zlib-compressed OTA uploads for ESP32. Should be pretty easy to adopt your project to use esp32-flashz. Let me know if you like the idea, I may come up with PoC PR or something. Cheers!
@vortigont Good Idea! That will be much appreciated and it will also ensure quick uploads for those who are experiencing upload issues due to signal loss.
I'd like to be able to upload cryptographic keys at the first installation. I don't have an exact workflow how this could be done, but on a conceptual level both encryption and signing would be nice.
Before uploading a new OTA file, the certificate has been deployed to the device as part of the initial installation.
ElegantOTA then require a file encrypted and/or signed with the private file. The device then cannot be updated without having access to the private certificate file.
@silverfisk That will be a nice addition as well.
We can use an approach with AES128-CBC or AES256-CBC encryption where we have a logic to buffer incoming chunks from the HTTP server and pass the buffer with correct length for decryption on-the-fly. As with CBC based encryption we already know the block size, we will be able to decrypt the file as we receive it from the user.
I'm only concerned about the performance drawback which will be incurred due to decryption, I hope it doesn't deter the HTTP upload process.
Very nice overall suggestions from the community! Love it.
If anybody knows a public key cipher which can work with parallel computing ( decrypting ), then please do let us know. I really prefer public key cryptography in IoT.
Till then, I only see AES to work as it's successfully implemented on the hardware level and in SDK.
asymmetric crypto is veeery slow, so it is used only for key exchange for symmetric ciphers. i.e. it makes no sense to encrypt/decrypt the firmware during upload and flashing. Only to sign the update and provide keys and certs.
@vortigont how slow are we talking?
At least signing would be nice to validate the binary. But if it's too slow to decrypt perhaps it's better to use HTTPS to encapsulate the payload in symmetric encryption? The binary will still contain the WiFi password and be treated as a secret though.
@silverfisk the difference is huge, 3-5 orders of magnitude or so AES
openssl speed aes-128-cbc
...
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
aes-128-cbc 1724778.47k 2039762.90k 2088857.86k 2083973.46k 2101630.29k 2093569.37k
RSA
openssl speed rsa2048
CPUINFO: OPENSSL_ia32cap=0x7ffaf3bfffebffff:0x18c05fdef3bfa7eb
sign verify sign/s verify/s
rsa 2048 bits 0.000239s 0.000015s 4183.0 68721.9
actually I do not know any real life use case for asymmetric crypto other than KEX or signatures.
HTTPS for fw transfer is definitely right approach to secure the transfer itself. Other than this Espressif has all the required features for signed fw in their SDK, no need to reinvent the wheel. I'm not that deep into this area, mostly use open source firmware's for my needs :)
@vortigont Nice info regarding ciphers. Let's see if we can get at least the firmware signing integrated. Espressif does provide features like SecureBoot V2 and encryption but for ESP32 only. I'm going to make this library supported on other platforms as well so it will be a very important feature to have.
@ayushsharma82 it's going to be a real pain to fit different arch'es into one lib, especially with things like encryption, lib dependencies, etc... You might wanna think twice about the design before stepping that sloppy road :) Might worth splitting the project into separate front-end/back-end parts. Well, anyway it's just a plans, isn't it? :)))
As for compression, I've done some dirty sketching and your front-end works pretty well with my flashz lib. Although there are some gaps that needs to be considered. Maybe open a discussion thread for this?
@vortigont You are pretty fast :)) I'll open a discussion thread. I did some renovations in past few days which switched the webpage from ancient 'Vuejs 2' to SvelteJS framework ( 73%~ size reduction - 54KB ( old ) to 15KB ( new ) ) and changed C++ library structure.
Do you want to wait for the changes to be live in a v3 branch? Or you can integrate it in the current version and I'll port it to V3 for you. ( whichever way you prefer )
Here's a sneak peak:
BTW size reduction is a must. 50k is too much nowadays for a simple upload form :)
I decided to merge AsyncWebServer support with V3. The only problem I'm facing is that the files are getting included irrespective of preprocessor conditions. Even when ELEGANTOTA_USE_ASYNC_SERVER is set to 0, the compiler/linker is grabbing async server header file.
@vortigont Can you take a look at v3-alpha branch to see if it's getting resolved by any means? ( Maybe my includes or structure is wrong ).
When lib is being build, preprocessor directives from the main sketch are not included. It could be handled pretty easy with global build_flags in platformio, but for Arduino IDE it does not work. This problem lasts for years already with no progress. Only this is enough reason to anyone not to use Arduino IDE, ever...
I actually never tried it in Arduino IDE for now. Tested with PIO and it's the same in there as well which is pretty weird. I used the reference from @vshymanskyy / TinyGSM lib which also uses preprocessor directives to include appropriate library. If it works in that case then it should work in ElegantOTA as well.
It works if you place all of your lib's code only in .h header files, i.e. no .cpp's. Headers are build on every inclusion and not processed if not included in any .cpp. Not an elegant solution, mostly used due to this ugly Arduino IDE heritage. With PIO you can use build_flags for the same purpose, no need to fit all code into .h, but in that case you will loose Arduino support. I can provide an example if you like.
@vortigont My javascript oriented mind struggles with C++ haha.
I've had some users pointing me out earlier when ElegantOTA had everything in a single header file, that lead to people unable include ElegantOTA header more than once ( for example usage in their own wrapper class ). Also Arduino compatibility is a must as apart from my personal interest, there are many users who use & prefer Arduino IDE.
Priority 1 is to make the v3 library compatible with Arduino IDE, then comes PIO ( which will most likely work as it is bound to be compatible with Arduino IDE )
I'll try some other way if the existing v3-alpha approach doesn't work out.
I don't know if this is needed, but reading the issues log on here, I found that the filesystem binary file for upload is supposed to be generated by the Arduino IDE (v1.8.19)compiler. When I compile for export on Debian Sid, I only get the NODEMCU binary. I can't find any spiffs binaries anywhere with Catfish. Maybe because SPIFFS is now deprecated? But LittleFS is overkill for this project and bogs me down. So if this is a real issue, and not just my ignorance, it would make sense to have a way to upload a tarball of the data folder that can be extracted to SPIFFS, or even a tool to upload files individually. EDIT! -- never mind, I found the SPIFFS.bin file location. Arduino IDE builds this in the /tmp folder on Debian. Not very nice of them IMHO.
@ayushsharma82 with Arduino IDE you are doomed to misery, sorry. And if you are going to support other MCU's and their dependent libs it's getting even worse. So, better plan your way in advance. Only 3 ways possible and I'm not sure which one is less ugly than the others :)
- all code in headers (and yes, you need to resolve issues with multiple inclusions)
- separate lib for every different arch/dependency
- teach users how to edit defines inside lib's .h files (or better to teach 'em using some other IDE/build system)
you can check this PR and it's descendants to find how deep this rabbit hole goes. Actually, I'm very skeptical about seeing this resolved any time soon.
I've gone through your code, can suggest some improvements besides flashz compression, but let's settle design questions before moving forward.
@NickelNuts abandon Arduino IDE and you'll get most of your issues resolved :)
Arduino officially dropped support for SPIFFS and switched to LittleFS. If you use PlatformIO on Debian for your project than building LittleFS image is as easy as running a command pio run -t buildfs and grabbing your littlefs.bin image file from .pio/build/* dir.
For individual file management I can suggest using ftp-lib
@vortigont Good info. I just checked Blynk library. They also support multiple platforms and in their case, instead of using preprocessor directives to select the appropriate platform, they rely on users to include their platform's header file ( which is somewhat acceptable in my sense ) For example: BlynkSimplyESP8266.h.
What do you think about this approach?
@vortigont Thanks, I have only been messing with these controllers for about a month or so, and have run into some stability issues with the Arduino IDE. I will look into PlatformIO. These projects I'm doing are not urgent, and mostly for my education, so learning another platform sounds like a good idea. That ftp server looks exactly what I need though
@ayushsharma82 looks like you are js-biased and do not get the idea of c/cpp build system (no offense, just explaining here) :))) Builder compiles .cpp's (and .ino) files and includes headers during compilation of those files. If you write a lib than all it's .cpp's are build separately of the user code without ANY idea of what is inside users files (defines, headers, includes etc...). If inside your lib's cpp you can set guards like #ifdef ESP8266 than you are good to go, because such flags are set by build system depending on selected tool-chain (i.e. board for Arduino IDE).
But if inside ANY of your code files you have same (or different methods/functions) depending on different libraries for the same platform (i.e. async and embedded arduino http-server) than you have no way to find out which one user is going to use! So, you must provide both dependent libs to be build to make linker happy afterwards.
If you put your lib code inside .h files than builder just ignores them until some cpp files includes this header and it is going to be a user code file, where he can define some tunable for his need that could be applied to your lib's code. But this introduces another issues with multiple inclusions and slow building (building same code multiple times).
Other build systems (PIO, Eclipse, CMAKE, etc) provide instruments for user to define global build flags that that are supplied to both user and lib code during build, that way he can configure lib build-time options to his needs.
@vortigont Thanks for the detailed explanation, this will certainly help me in the long run 👍🏻.
If inside your lib's cpp you can set guards like #ifdef ESP8266 than you are good to go
I tried this and it worked perfectly :) we can use a combo of flags set by build system and custom definitions like ELEGANTOTA_USE_ASYNC_SERVER. I don't expect anybody to run both types of server simultaneously in the same code ( even if they do, that will probably crash the ESP as these both don't play well together ).
I've gone through your code, can suggest some improvements besides flashz compression
What else would you like to suggest for the v3-alpha branch?
I meant some code optimizations here and there. As a new features it would be nice to have http-client update integrated also. Like you provide URL to the firmware image via WebUI and MCU downloads and flash the image itself. Something like in Tasmota's update, etc... I have this feature in flashz lib already, but without fancy UI :)
You may have already spotted bits of it in the v3 branch :) , I'm working on ElegantOTA Cloud ( almost complete ) which is much secure than hosting the firmware on your own.
The ElegantOTA Cloud will feature:
- TLS connection ( cert management will be done automatically )
- Push updates
- A dashboard to manage all the connected devices along with device status
- Approximate device location on map based on IP address
- 'Cluster' support in which you can have a single firmware source for multiple devices.
This is geared towards businesses but I'll have a forever free plan for individuals so that they can benefit from the platform for their own projects/hobby.
it is possible to return a special error if it does not have enough memory to upload a file ?
Maybe my question is invalid, probably because I'm an amateur. However, how can we add other html pages to this project? Example:
Access an HTML "index" on esp32's ip, without the need for a my_ip/update
The second question is: What is the difference between Elegant Ota and AsyncElegantOta?
