Migrate from BoardConfig.h to UF2 patching for releases

[FeralAI]

Issue

As the project grows in popularity it's becoming more of a challenge to validate and maintain separate configurations for each unique type of board and controller available. The current process has several drawbacks:

• Unclear to users which release UF2 to use
• Arbitrary and manual process to validate, approve, and maintain each custom configuration
• Makers want an easy way to configure the firmware for their projects and products
• When a board has a unique pin layout it requires BoardConfig.h and build configuration
  • If BoardConfig.h is not fully populated the build will break for that board
  • When a new firmware option is defined, all BoardConfig.h files require an update
• Each board configuration requires a uniquely configured build task to generate the unique UF2 file
• Each unique build takes a measurable amount of time to run, extending the overall build time significantly

Some of these issues, like missing BoardConfig.h params, can be addressed out of scope of this proposal. Those updates can and should be considered separately as well.

Proposal

@mthiesen, @deeebug and I have discussed a technical approach to address these issues using a post-build UF2 patching process. The high level steps are:

• The build process is stripped down to a single, base UF2 file likely targeting the Raspberry Pi Pico
• An amount of flash space is reserved at a specified flash address for a base configuration file
• Build process modified to patch UF2 file's base configuration area
• When GP2040-CE starts, if FlashPROM area is empty or invalid, read base configuration file and save to FlashPROM as defaults

UF2 Patching

This general approach to UF2 patching has been determined to be viable:

• Acquire the compiled GP2040-CE .bin file, either via:
  • Excluding the UF2 generation step of the build process
  • Utilize a single build configuration and patch the UF2 file
• Patch the .bin file at the specified base configuration flash address
• Pack .bin as UF2 file and rename for that configuration

This will require a bit of custom scripting of the build process. See https://github.com/microsoft/uf2/blob/master/utils/uf2conv.md for details on packing and unpacking UF2 files.

There has also been discussion around generating a single UF2 and using that as the base build for an online patcher.

Firmware Updates

A standardized board config format will be defined for the new configuration method. We anticipate using the Protobuf work by @mthiesen for this. The Protobuf feature includes JSON serialization and deserialization and seems to be a logical fit.

Regarding BoardConfig.h parameters, these will all likely migrate to properties in the Protobuf schema and will require sane default values when not present in the base configuration.

Releases

There are concerns relating to the user experience when trying to download and configure GP2040-CE from the releases area. This is being addressed separately via a visual download page and trimming down the list of preconfigured UF2s that are offered.

Another idea being floated is to have a UF2 patching page where makers and advanced users can upload their configuration file and have a patched UF2 output to them. This in theory could all be handled client-side to avoid the requirement of a dedicated host for patching and hosting files.

[bsstephan]

Per recent conversation and based on my tinkering with gp2040ce-binary-tools, I am going to assign this to myself. It will take some time, I am sure, but I believe I can achieve all the design above.

[bsstephan]

Summary of where we are now/what we need to do:

Config work

1. clean up BoardConfig.h files [in progress by @TheTrainGoes]
2. convert foo/BoardConfig.h to foo-config.json [I'm waiting for 1]
   1. I will probably do this by loading the cleaned binaries onto a nuked board, letting the firmware convert them to protobuf, and then gp2040ce-binary-tools them to JSON

Firmware work

1. implement a new reserved portion of flash for the default, read-only board config [sketched out locally]
2. abandon legacy config to protobuf conversions [done in a branch]
3. abandon BoardConfig.h to protobuf conversions [done in a branch]
4. read board config protobuf if user config protobuf is not present [sketched out, to implement, pretty much just a copy of how the user config is loaded but with a different memory address]

Build work

1. feature to have gp2040ce-binary-tools concatenate a .bin and a .json and populate the flash area with the board default in a new .bin [to implement, up next]
2. feature to have gp2040ce-binary-tools convert a .bin to a .uf2 [to implement]
3. have GP2040-CE build one generic .bin rather than per-board [I'm currently cutting stuff out of the build in a branch]
4. take the generic binary and write each JSON variant to a new UF2 [to do]
5. finish cleaning up dead code now that there's no initial import/migration [wrapping up]