ducalex
Add self-updater for devices with no .fw support
Hi
I understand that the Odroid and G32 devices allow onboard updating via the .fw firmware files, are other devices able to update in such a way?
For instance the VMU and Null Nano share the Odroid reference design mostly so I had assumed that they would be able to update via this method also? I was kinda hoping as both are not easy to flash since they have no onboard flash circuitry :)
I hadnt really looked into it too much as they were not part of an official release (although there were no official .fw files compiled, so not sure if thats an indication or not!)
I have compiled my own .fw files and on the VMU for instance, I placed it in /vmu/firmware. Am I correct that I should be able to turn on with a button held and in the recovery menu get an 'reboot to updater' option? I dont get that option for some reason. I can connect to the internet, download the latest vmu.img file and I noticed that does get correctly placed into the vmu/firmware directory but a reboot didnt work.
I've spent some time comparing odroid target files and the main config.h and I cant see anywhere that may cause a difference.
Apologies if I misunderstood and that method is only available on the 2 above devices maybe due to some additional onboard hardware.
Thanks
P.S. on the 1.45 changelog there were some missing updates :)
New device support: VMU New device support: CrokPocket by Megazoid Added support for additional GPIO Extenders: MCP23017 & PCF8575
Hey Dyna, You new a special firmware on your device to "install" retro-go's .fw files. Here is the link to ducalex's version of it: https://github.com/ducalex/odroid-go-multi-firmware It basically allows you to install multiple apps (such as retro-go's) on your device as the name implies
Hi
Oh yeah, I understand now. I wasnt aware of this before now! I never had a Odroid Go but do kinda remember seeing videos people loading apps in a similar way, so must had been that!
Looks like it could maybe be adapted for the VMU etc, assuming the display driver works as thats pretty much the only main difference compared to the original Odroid. Sounds like a future project for a (very) rainy day :)
It certainly would be nice to have retro-go be able to update itself, instead of relying on odroid-go-firmware/odroid-go-multi-firmware/esplay-base-firmware.
tomvanbraeckel added a flasher to the launcher for fri3d-2024 but it only worked on the S3, so it was never added to retro-go. For the regular esp32 the flasher pretty much has to be an additional app, as we can't flash over a running app (ie the launcher could update emulators but not itself). Which means retro-go will have to be 300-400KB larger, going over my beloved 4MB goal. That is why I never added the feature to retro-go, even though I did write a flasher for it at some point.
But I'm less concerned with flash size now (with fmsx we're already over 4MB), so I'd be open to adding a flash utility to retro-go now. I'll see if I can find the one I wrote years ago!
Edit: Looks like whilst discussing with tom in #138 I did write a proof of concept https://github.com/ducalex/retro-go/commit/6ec06678b61dd7084f09fbe11400adfedff2e333. This is definitely not good enough, though.
That would be awesome if its possible, I'd guess most devices would have 16MB to play with and those devices with 4MB can choose which they do and dont want to compile, which like you said is already the case with MSX.
Fully OTA updates would be pretty cool
I've pushed the first working prototype to the dev branch!
To test if you need to do a few things:
Make sure your target's config.h has these:
#define RG_UPDATER_ENABLE 1
#define RG_UPDATER_APPLICATION "updater" // <--- Specifically this
#define RG_UPDATER_DOWNLOAD_LOCATION RG_STORAGE_ROOT "/nullnano/firmware"
Then add "updater" to your target's env.py DEFAULT_APPS:
- You can either list all the apps you want:
DEFAULT_APPS = "launcher ... updater" - Or you can just add updater to the default value:
DEFAULT_APPS = f"{DEFAULT_APPS} updater"
Then build a new full image and flash it through serial.
The updater is accessible through the launcher's about menu.
Several limitations remain:
- The updater will ignore new or bigger partitions in the .img file
- If the updater fails during the launcher update, there will be no way of going back in it so the device will be "bricked" until you flash over serial.
- No integrity check (but I do embed a CRC32 in .img now)
- The progress display is a bit messy with big font/small screen
- No rollback mechanism. It would be nice to add but will "waste" 2MB of reserved flash
- No .fw file support, only .img. I think going forward standardizing on plain .img is better, but I'll consider adding .fw support
- The file picker can't browse outside the dedicated folder, not super user-friendly
- Probably tons more...
All fixable issues that I have already overcome in my other project, odroid-go-multi-firmware, but I think I'll wait for the updater to work reliably before working on these issues.
That sounds awesome, I won't be able to test for a week or so as I'm away from home currently but I'll look forward to testing.
Few thoughts and this might be covered and I'd normally scan the code to try to understand first but not doing that on my phone 😁 you mentioned about issues due to partition sizes, does it currently have ample space for the current apps in that partition to grow? I don't know how close they are or how much additional space maybe required etc, e.g would changing to a few IDF version increase the size? Just thinking that giving extra space at this stage may help with seamless future updates if there big changes to certain apps while keeping the 4mb limit in mind but a much bigger tolerance for 16mb devices.
Hi @ducalex , The updater would be really nice to have! I don't know if this is possible or a viable solution, but maybe merging all emulators into a single partition could be a bit simpler for updating, like instead of having to leave headroom for gwenesis, retro-core, pr-boom-go.... We would just have a big "emulator" partition taking up all the space left after launcher and updater + headroom. Also it would be possible to update the firmware even if an emulator is added. But I was wondering if there was a reason for some emulators being separate apps?
does it currently have ample space for the current apps in that partition to grow? I
The short answer is no. I've increased the partition sizes a handful of times over the years, but they're still quite full. (My mkfw.py/build-fw gives you that info neatly but unfortunately build-img doesn't, I should add it there.)
Current usage
[0]: type=0, subtype=16, size=1048576 (89% used), label=launcher
[1]: type=0, subtype=17, size=524288 (57% used), label=updater
[2]: type=0, subtype=18, size=1048576 (92% used), label=retro-core
[3]: type=0, subtype=19, size=851968 (90% used), label=prboom-go
[4]: type=0, subtype=20, size=1048576 (94% used), label=gwenesis
[5]: type=0, subtype=21, size=655360 (89% used), label=fmsx
[6]: type=0, subtype=22, size=851968 (72% used), label=gbsp
I agree that if a device has 16MB and retro-go is the only thing there, the rest is wasted so we might as well future-proof.
However as the maintainer I have to balance many things. Some devices share that 16MB with other things than retro-go. I also need to make sure that a reasonable subset can always fit on 4MB devices.
Ultimately I don't know what the best solution is... Many options could work, but they all have downsides too. Having targets define their own partition table. Or having targets define the flash size they want to give retro-go so that rg_tool.py could automatically apply some extra padding if possible. Or just raise the partitions now and say "full retro-go is 8MB now, deal with it 🤷♀️".
I don't know if this is possible or a viable solution, but maybe merging all emulators into a single partition could be a bit simpler for updating
That's the dream, isn't it? No more wasted flash for 12 copies of esp-idf, wifi drivers, or even libretro-go!
But I was wondering if there was a reason for some emulators being separate apps?
All programs reserves a certain amount of memory (static variables, globals) unless they're designed to be fully dynamic (which in C is quite rare). This memory is limited on esp32 (~200KB) and reserved at compile/link time, so in retro-core memory reserved by the NES component will never be usable by the PCE component even if NES isn't running. I've had to make substantial changes to all the apps in retro-core to make it fit (convert some of that to dynamic allocations, refactor to get rid of others) but even then all those apps still "rob" the others from 5-10KB (up to 25KB if you count IRAM) of internal memory each.
A similar issue exists with the IRAM segment, some apps use it to speed up some functions (search IRAM_ATTR in the codebase). That memory is limited and reserved at compile time also.
Since we can guarantee that only one component will run at a time, someone smarter than me might be able to come up with a custom linker script to allow BSS (but probably not IRAM) segments to overlap between specific components so that they can be compiled together without having to change all the code. But alas, I am not that person.
You can find memory usage of components in an app by running idf size-components. Here's for example a truncated output for retro-core:
Total sizes:
Used static DRAM: 37188 bytes ( 143548 remain, 20.6% used)
.data size: 6836 bytes
.bss size: 30352 bytes
Used static IRAM: 108586 bytes ( 22486 remain, 82.8% used)
.text size: 107559 bytes
.vectors size: 1027 bytes
Used Flash size : 850787 bytes
.text : 639543 bytes
.rodata : 210988 bytes
Total image size: 966209 bytes (.bin may be padded larger)
Per-archive contributions to ELF file:
Archive File DRAM .data & 0.bss .rtc_noinit IRAM0 .text & 0.vectors ram_st_total Flash .text & .rodata & .appdesc flash_total
libsnes9x.a 4 7267 0 0 0 7271 189618 26202 0 215824
libnofrendo.a 4 5406 0 19978 0 25388 40167 64086 0 124235
libretro-go.a 13 3129 2540 3089 0 8771 69480 39243 0 114365
libhandy.a 8 108 0 0 0 116 73327 5727 0 79062
libsmsplus.a 8 3953 0 15032 0 18993 41716 20241 0 76997
libpce-go.a 12 4300 0 0 0 4312 46787 3435 0 50234
libgnuboy.a 0 1673 0 15897 0 17570 20514 9101 0 45512
libgw-emulator.a 0 1497 0 0 0 1497 17080 2269 0 19349
Finally had a chance to test this after my hols. Pulled the latest dev and flashed, all seems to be fine. Reflashed the exact same img without issues, it did warn me my devices didnt match but that might had been the way I named it. Interface looked fine to me, maybe not the most pretty but I'd taking working good over looking good :)
I did then recompile but adding gbsp, it reflashed all the other partitions but didnt add gpsp, which I kinda expected but on the plus side it didnt break anything, successfully reflashed the existing ones and worked fine.
The current dev (standalone SNES) and also updater does increase the build size quite a bit, my CYD device only has 4MB flash, previously I could fit everything except MSX however now that SNES is standalone, its increased the size. Currently launcher, retro-core, prboom-go, gwenesis fits, but adding updater wont, also adding SNES wont either, I'd need to remove doom or MD to add either SNES or updater. Not complaining :) just advising.
Just throwing it out there, would it be possible to leave SNES inside retro-core, maybe as SNES Legacy but still have Snes9x standalone for new development for faster chips? I know that doesnt resolve the updater needing an additional package to be removed but at least it only feels like we're losing one :)
All that being said, the new updater is great and will help make future updates a more seamless
The updater will soon be able to edit the partition table and flash pretty much anything except itself or the bootloader (to reduce the odds of soft-bricking in case of failure).
and also updater does increase the build size quite a bit, my CYD device only has 4MB flash, previously I could fit everything except MSX however now that SNES is standalone
The final updater will be smaller but yeah adding it to your CYD probably means keeping only one of: gwenesis, snes9x, fmsx, gbsp. Even if snes9x stayed in retro-core you probably wouldn't have room to fit any of the others unfortunately.
One thing I'm still considering is embedding the updater into each individual app (into the recovery mode). The launcher could then update everything except itself, then reboot to another app that enters the updater again to update the launcher. It would save maybe 200-400KB of flash. But it remains to be seen if it's worth the complexity.
As for standalone snes9x:
I agree the standalone snes9x isn't ideal, but my decision wasn't only to help the P4 port. Snes9x on its own is simply faster. It always was, but I had given up getting it to run full speed so the space savings were worth the bundling at the time. Because hey, it wasn't playable anyhow! But now with the overclock and audio task it's almost playable on esp32. So I'll keep it as standalone but I'll see if I can reduce its size a bit.
By the way I've rewritten the .img generator, now it shows you nice stats like for .fw:
Partitions:
[0]: type=0, subtype=16, size=1048576 (89% used), label=launcher
[1]: type=0, subtype=17, size=524288 (57% used), label=updater
[2]: type=0, subtype=18, size=851968 (87% used), label=retro-core
[3]: type=0, subtype=19, size=851968 (90% used), label=prboom-go
[4]: type=0, subtype=20, size=655360 (84% used), label=snes9x
[5]: type=0, subtype=21, size=1048576 (94% used), label=gwenesis
[6]: type=0, subtype=22, size=655360 (89% used), label=fmsx
[7]: type=0, subtype=23, size=851968 (72% used), label=gbsp
Flash size: 6.250 MB
File 'retro-go_1.46-dev-45-gf47ff-dirty_odroid-go.img' successfully created.
Be careful if you try it, I haven't tested it very much yet.
Looks like some great improvements to the updater!
Its cool about the CYD and stuff, at the time I was more thinking if the updater was excluded it would had still been nice to be able to keep the same packages as v1.45 but there comes a time where you cant keep making comprises for the sake of the odd device, as long as core and launcher fits then that covers most of the major systems imo.
If having SNES standalone makes it faster then its certainly worth the extra space and like you said its getting very close to fully playable now!
I understand your disappointment entirely, trying to fit as much as possible in 4MB was always a main goal of retro-go and I spent so much (enjoyable) time experimenting with ways to achieve it!
There is still https://github.com/ducalex/retro-go/issues/167 that I'm hopeful to achieve one day. I almost did it earlier this year but I had to undo it when I found performance issues in some specific SMS games, where the standalone retro-core normally never frame skips more than 1.
Now that snes9x is no longer part of retro-core I will probably give launcher-retro-core another shot, it might be just enough freed RAM to make it work.
I was thinking the other day that fitting everything into the space, speed and memory constraints of a ESP32 must be pretty challenging. When I built up my first RetroGo device (Null Nano) probably the first thing that stood out was how great the software side was, felt like something from a much more power device. Everything was very integrated and felt seamless, something I wasnt really expecting from such a low powered device if I'm honest.
I guess having more standalone cores/apps etc means duplicating a lot of the core functionality? Which takes up additional space duplicating it. Interesting. Honestly I'm not too worried about the CYD, I knew going in it was gonna be limited, strangely its been a pretty popular project but if needed it can be left with v1.45.
I have been working on an updated VMU with an S3 so at some stage I may do a pull request to update that as long as the new revision works as intended.
I guess having more standalone cores/apps etc means duplicating a lot of the core functionality? Which takes up additional space duplicating it
Yes. For an example with snes9x, the binary is 540KB of which only 210KB is snes9x, the rest is duplicated in all apps:
- 110KB is retro-go
- 220KB is esp-idf
Wifi, when enabled, adds another 400KB per app. Bluetooth adds even more.
I knew going in it was gonna be limited, strangely its been a pretty popular project but if needed it can be left with v1.45.
I will probably extract only the bug fixes from the dev branch and do a 1.45.1 before I do a 1.46, so you'll have that at least!