InfiniTime
InfiniTime copied to clipboard
InfiniTimeOrg
External flash usage
It would be good to map out where LittleFS would potentially be used in InfiniTime, right now and in the future. What would have to be migrated and what can't be migrated.
Current usages of flash contain:
- [x] Bootloader scratch sections
- [x] Settings storage
- [x] Datetime backup (https://github.com/JF002/InfiniTime/issues/87)
Future usages might contain:
- [ ] Resource storage (for LVGL)
- [ ] Buffer for HR/step data (https://github.com/JF002/InfiniTime/issues/493)
- [ ] ??? Please leave your comment below
I managed to do a first performance test between RAWFS versus LITTLEFS. The use of LITTLEFS adds another 20Kb in the firmware. As we can see in the video the performance between the two is evident.
The LITTLEFS introduces some advantages in the management of the files, but I think that the loss of performance and increase of the firmware does not justify the gain of functionalities.
In my case, I will not invest more time in the littlefs solution, I will try to improve RAWFS and add some features to allow partial updates or subdivide between resources and watch faces, so that I can have customizable watch faces without having to always send everything.
What's the LittleFS configuration you used?
https://github.com/joaquimorg/PinetimeLite/blob/1db8712ecd49ecb2ee37147eebe2e5b6dac521ac/src/components/fs/FS.cpp#L81
I will try to optimize the settings, because I think better read performance is doable and worth the investment.
The advantages we'd get from LittleFS, such as reduced maintenance and development burden, full-flash wear leveling, better failure tolerance (CoW), thought-out and documented design, higher interoperability and alternative implementations are more than some.
In my tests I had already identified that LVGL is always opening files to read, and did not keep the files open while refreshing the screen. After all, it was my mistake, the code allow that there could be several files open at the same time, but LVGL does not take advantage of this if it is not indicated in LV_IMG_CACHE_DEF_SIZE how many images we want to keep open. I will make this change and carry out further tests.
I would also prefer to use littlefs, but I don't want its use to translate into having something that makes a point of losing the little performance we have.
When I indicated that I would not invest more time in littlefs it was in the continuation of developing the mechanisms so that we can have a management of the files, delete files, create folders etc...
So whenever I discover something that can improve the performance I will test it.
Thanks @joaquimorg and @Avamander , great work!
From your video, the performance penality looks... surprisingly huge! Maybe @Avamander will be able to fine-tune/optimise the integration of littlefs with lvgl?
Now... Maybe we are trying to solve many problems with a single solution? The use-case tested by @joaquimorg on his video is a "read-only" use case : we store files in the external memory once and only read them from the firmware. Those data are (over)written only when upgrading the graphical assets. The use-case where a more advanced file system like littlefs would bring a lot of advantages are "read/write" use-cases like storing 24h or heart rate data, user settings, save the master piece the user's just drawn using InfiniPaint,...
Maybe we can explore the possibility to split the external flash space available in 2 parts : one read only to store fonts, icons, bitmaps, static assets. And the other one to store dynamic data that is often read/written.
What's your opinion?
Thanks joaquimorg and Avamander , great work! Maybe Avamander will be able to fine-tune/optimise the integration of littlefs with lvgl?
joaquimorg deserves more credit here and he also said that LV_IMG_CACHE_DEF_SIZE might provide a quick and easy way of regaining the lost performance, if that doesn't work out I'll investigate as well.
Maybe we can explore the possibility to split the external flash space available in 2 parts : one read only to store fonts, icons, bitmaps, static assets. And the other one to store dynamic data that is often read/written.
Yes, that's a theoretical possibility. But the returns are somewhat smaller. If it still turns out that LFS is not worth it, we just have to go with an alternative solution.
Maybe we can explore the possibility to split the external flash space available in 2 parts : one read only to store fonts, icons, bitmaps, static assets. And the other one to store dynamic data that is often read/written.
I think it may be a viable solution, a few days ago we were discussing this on the discord.
Really the part of the resources for the firmware can be read-only and we don't have the need to have littlefs for that, and we can have an r/w zone with littlefs, in total we have about 3.4 Mb still free if we share between the two we have space for lots of applications to save data.
In the end we have to remember that we are talking about a device that has its limitations and we do not want to have a very complex system, my goal is always to look for a way to try to do things to get the maximum performance from the smartwatch which it often leads to doing things in a less generic way, more oriented towards the optimization of the few resources we have.
This weekend I will try to do some more tests to try to see if the changes have influence.
Maybe we can explore the possibility to split the external flash space available in 2 parts : one read only to store fonts, icons, bitmaps, static assets. And the other one to store dynamic data that is often read/written.
We can also make a proper filesystem with folders (Directory's) , Example: A fonts folder for the font files , Assets that have all assets for the firmware (Resources) , AppData (Contains appdata that is stored to the spi flash) , Userdata (User Settings , Time Settings etc,)
If possible we could also try to make the Assets r/w , so that the user can add or modify the assets without having to make an entirely new firmware....
This could be a vulnerability but a hierarchy system can be put in place: System > User > Apps. This might need a lot of code so might be out of reach.
I did some more tests changing the LVGL configuration (LV_IMG_CACHE_DEF_SIZE) and it really made all the difference, of course there is some loss of performance, but I think it is possible to live with it. Now the hardest work begins, turning everything into a PR ...
This is really impressive! Thanks for your experimentation!
So, LVGL could be the unique answer for all our "external" storage use-cases?
For the PR, what would be the easiest way for you? A single PR of split the changes accross multiple PR to ease the review process? Do you have a branch on github so we can have a look at how it works?
Do you have a branch on github so we can have a look at how it works?
The branch with the test code is here https://github.com/joaquimorg/PinetimeLite/tree/UsingLITTLEFS
For the PR, what would be the easiest way for you? A single PR of split the changes accross multiple PR to ease the review process?
I will try to split as much as possible to be easier to include without causing breaks in what already exists, and thus also be easier to review, however it will only work in full when everything is together.
I will try to split as much as possible to be easier to include without causing breaks in what already exists, and thus also be easier to review, however it will only work in full when everything is together.
Awesome! Feel free to comment here or to ping us in the chat if you need help!
I did some more tests changing the LVGL configuration (LV_IMG_CACHE_DEF_SIZE) and it really made all the difference, of course there is some loss of performance, but I think it is possible to live with it. Now the hardest work begins, turning everything into a PR ...
Awesome! Really nice that it has possibilities to be speedy.
@Avamander provided this link in the chat room, it could be interesting to have a closer look at this lib/protocol : https://github.com/adafruit/adafruit_circuitpython_ble_file_transfer
@joaquimorg Any news on this topic?
I was thinking : how will we use this filesystem? I mean, we have multiple types of data :
- read only data for the firmware (I think you call them resources in your FW): logo, icons, pictures,...
- User settings
- "run time data" like HR and steps values
I have many questions about this, and I think you've already solved most of them in PineTimeLite :)
User settings and runtime data will be created and read/written by the firmware.
I'm more concerned about the resources : The user will need to manually flash (OTA) these data for the FW to work correctly. What happens if the user does not send the file? Is there an error? Or a message asking the user to flash the data? Or the FW works in "degraded" mode without the nice pictures?
Also, what protocol do we use to send the resources? DFU? A custom one? Another one (like the one from adafruit mentioned by Avamander) ?
Next : how do we generate these data? I guess you have a script that converts pictures into that binary file? Do you also handle versioning so that the firmware can check if the correct resources file is installed ?
These questions are mostly implementations details, but I think there are many many ways to do all of this, and I would like to have your opinions about this :)
@joaquimorg Any news on this topic?
I was thinking : how will we use this filesystem? I mean, we have multiple types of data :
- read only data for the firmware (I think you call them resources in your FW): logo, icons, pictures,...
- User settings
- "run time data" like HR and steps values
I have many questions about this, and I think you've already solved most of them in PineTimeLite :)
User settings and runtime data will be created and read/written by the firmware.
I'm more concerned about the resources : The user will need to manually flash (OTA) these data for the FW to work correctly. What happens if the user does not send the file? Is there an error? Or a message asking the user to flash the data? Or the FW works in "degraded" mode without the nice pictures?
Also, what protocol do we use to send the resources? DFU? A custom one? Another one (like the one from adafruit mentioned by Avamander) ?
Next : how do we generate these data? I guess you have a script that converts pictures into that binary file? Do you also handle versioning so that the firmware can check if the correct resources file is installed ?
These questions are mostly implementations details, but I think there are many many ways to do all of this, and I would like to have your opinions about this :)
It could show an error saying "Resources not found" similar to the mi band that says "Please connect to mi fit"
We could use the same method we are using now for this , the resources can be a compressed file that is uncompressed once sent to the watch.
@joaquimorg Any news on this topic?
Hi, unfortunately I haven't been able to dedicate much time to the project, my work has taken up a lot of time.
I was thinking : how will we use this filesystem? I mean, we have multiple types of data :
- read only data for the firmware (I think you call them resources in your FW): logo, icons, pictures,...
- User settings
- "run time data" like HR and steps values
I have many questions about this, and I think you've already solved most of them in PineTimeLite :)
User settings and runtime data will be created and read/written by the firmware.
I'm more concerned about the resources : The user will need to manually flash (OTA) these data for the FW to work correctly. What happens if the user does not send the file? Is there an error? Or a message asking the user to flash the data? Or the FW works in "degraded" mode without the nice pictures?
In my fork I do not present any information, what happens is that the images do not appear, however it would be good to present some information about the lack of updating the resources, to avoid that the user does not think that the update was not done correctly.
Also, what protocol do we use to send the resources? DFU? A custom one? Another one (like the one from adafruit mentioned by Avamander) ?
I do not think that DFU can be used, since it only allows to have the Bootloader, Softdevice and Application in the payload, so I was able to investigate. So we would have to have something to be able to upload the resources. It would be useful to be able to upload together or separately.
The adafruit example is very similar to the one I have, mine is a little simpler but it works. If we want to have a way to be able to control the files that are in the flash we will have to have an additional service to the DFU, in order to be able to list, create and delete files.
Next : how do we generate these data? I guess you have a script that converts pictures into that binary file? Do you also handle versioning so that the firmware can check if the correct resources file is installed ?
Yes I have a script that converts PNG to LVGL images, then another script creates the RAWFS file, it is this file that will be sent to the watch. And yes the RAWFS has the version in the header, to be able to validate that it is the right one when the watch starts.
However with littlefs I think it cannot be done the way I have it, in the tests I did the files were all separate, and this is the way that makes the most sense to me, for example having a "resources" folder and inside the necessary images, there will also be a metadata file in that folder with the version information and other information that is necessary to identify that the resources are valid for the version that the watch is running.
The generation of resources can be done by generating a zip with all the necessary files, and it is on the side of the companion application to open that zip and send each file to the watch inside the "resources" folder.
The generation of resources can be done by generating a zip with all the necessary files, and it is on the side of the companion application to open that zip and send each file to the watch inside the "resources" folder.
Much better than my idea of senting a compressed file , will be less taxing on us and the watch.
@joaquimorg Thanks for all these informations, and I fully you prioritize your job on this project :)
I don't think we have a use-case for a "file explorer" API right now but we might need a way to remove older resources when we upgrade to a new version to avoid filling the memory with data that are not needed anymore.
I do not think that DFU can be used, since it only allows to have the Bootloader, Softdevice and Application in the payload
Technically, we might be able to use the DFU protocol, as the field for the data type is 1 byte long and only 3 values are defined by NRF (other values are "reserved"), so we could use those reserved values to add the resource type, but that wouldn't be "nrf" compliant. But, as you said, dfu does not provide any way to control the files (to delete older/unused files, for example).
So, if I understand correctly, we have to implement
- the filesystem
- the protocol to send data to this filesystem
- the file format of the resources + versioning
- using those resources in the firmware and handling the case when the resources are not available.
Base support for littlefs added in https://github.com/JF002/InfiniTime/pull/438
- [x] the filesystem
- [ ] the protocol to send data to this filesystem
- [ ] the file format of the resources + versioning
- [ ] using those resources in the firmware and handling the case when the resources are not available.
It would be good to map out where LittleFS would potentially be used in InfiniTime [...] * [ ] ??? Please leave your comment below
Replace the manual FW validation with an automatic one:
- store a failed-boot-counter
- clear it after a successful boot
- increment it as early as possible in the bootprocess
- roll back when the counter exceeds some threshold, e.g. 3(*)
So, if the FW update is not good, it would try to start, fail three times and then roll back.
No need for the user to know about and understand the manual validation and dig around in the settings to find the place.
(*) Don't rollback after only 1 failed boot. Someone, sometime will turn it off right after turning on, or it will fail due to low battery. That doesn't mean the FW should be rolled back
@doniks actually there is a need, it's verifying the hardware works. That can't be done software-side. There's also no need to dig if the quick start guide has been read.
@doniks actually there is a need, it's verifying the hardware works. That can't be done software-side. There's also no need to dig if the quick start guide has been read.
When you say whether "the hardware works" do you actually mean whether the hardware itself broke for some reason? I guess not, because actually if the hardware broke, then rolling back the FW won't help either. So, I assume we are talking about "hardware support", ie, some firmware issue.
I maintain that it's a better user experience to only roll back if the FW can't run at all, ie the device is a brick unless we roll back. As long as the FW does start (and the user is able to perform another update/downgrade to some other firmware version) then I think it's a better user experience to keep the new version by default, rather than to roll back by default. Even if the new FW has some serious issue like it can't read the accelerometer anymore. You wouldn't assume that every update is broken unless the user explicitly confirms that this time around it's ok. What must be prevented is that some bad update leads to endless boot-crash cycles and 'normal' users are unable to fix it. But once you have avoided that, you wouldn't want to push too much 'maintenance' hassle onto the user
Base support for littlefs added in #438
- [x] the filesystem
- [ ] the protocol to send data to this filesystem
- [ ] the file format of the resources + versioning
- [ ] using those resources in the firmware and handling the case when the resources are not available.
If I investigated correctly, #756 should be the second bullet point, right? I would like to help working on the next step to get the external memory usable for resources, but I just wanted to ensure I didn't overlook some work that has already been put into specifying the file format / directory structure for the resources. If no one objects, I would open a new issue with some proposals.
Just as a recap, what kind of resources do you want to support? I can think of:
- fonts, potentially somehow subdivided to support different special characters (see #212 or #388)
- images (to support watch faces similar to the ones in https://github.com/joaquimorg/PinetimeLite)
- strings for internationalization (see #363)
You are right, #756 is the 2nd bullet! So now, we have the filesystem and an API to send data to this file system from a companion app. We currently haven't specified any file format or directory structure so your analysis and proposals are more than welcome!
The list of resources you mentioned looks quite right. Note that we don't need to support all of them in one shot. We can start with only one type of data, the one we'll find the easiest to implement, and add the other ones later one.
For example, for the December Pine64 community update, I (with the help of @geekbozu) wrote a very simple change in the Digital watch face to display a background image if it's available on the file system. I think that would be a nice first step to implement.
Note that we'll probably want to add a bullet point : "improve the performance of the SpiNorFlash driver and/or fileesystem integration", as reading a full picture from the filesystem slows down the whole UI.
So something to note, A lot of the slowdown has to do with how often LVGL opens the image and how much of it it can cache. Since LVGL is "blitting" the image it has to read the image and seek in it every line...
A large speedup will be using properly "compressed" (Smaller Bit per pixel with pallet data) images on the SPI flash instead of the demo we put together which loaded uncompressed images.
That needs a LVGL Image Decoder To handle this "compressed" format.
I've finally decided to focus on this feature. I've created a new project dedicated to this topic.
First, a did a few benchmark of a first use-case : read a picture and use it as a full screen background for the digital watchface.
First, using InfiniTime without any change to the FS layer. Vertical scroll looks good, and similar to @joaqimorg results on this video. However, left/right animation are veeeery slow.
https://user-images.githubusercontent.com/2261652/169712859-54941329-9395-4680-804c-2a22f4f2a6f4.mp4
I compared my results with PineTimeLite, @joaquimorg 's fork. Both animations are very fast. This implementation uses a "raw" FS layer instead of LittleFS.
https://user-images.githubusercontent.com/2261652/169712700-5cca2ccb-0148-4453-8d92-4e31f2353199.mp4
I integrated this raw fs layer in InfiniTime and observed similar results:
https://user-images.githubusercontent.com/2261652/169712687-a18e54c6-4107-48e9-aafb-5fdd571145d7.mp4
So, my results are consistent with the ones from @joaquimorg, great!
The current implementation with LittleFS (which is still our preferred one) is quite efficient for linear readings, but way too slow for random read accesses needed by the left/right animation. During the left/right animation, LFS seeks in the file in the correct positions and then reads 8 bytes. This translates to 2 to 8 read accesses on the SPI bus. I think this is LittleFS that is looking for the block that contains the data and this is probably what's causing this huge slowdown...
Next, I'll try other use-cases : read smaller pictures and icons, and also fonts. The goal is to check which use-cases could already be implemented and which ones need more work!