GP2040-CE icon indicating copy to clipboard operation
GP2040-CE copied to clipboard

Start migrating to protobuf board configs

Open bsstephan opened this issue 10 months ago • 6 comments

This adds a post-build CMake command (thanks @FeralAI!) which uses gp2040ce-binary-tools to write a Protobuf board config into the ~~.bin and~~ .uf2 files generated by the normal build.

A Protobuf board config is a second reserved space at the end of the flash, essentially the exact same idea as the user config, just in an adjacent block of space. It is used to replace the Protobuf defaults with specified values (for now, as a demo, just the boardVersion version number) from the build, in order to convey a specific configuration with less C++ code in initUnset....

For the moment this doesn't undo anything material in initUnset..., but with this in place, the next step will be setting defaults in the config.proto file and overriding them with specific board config JSON files (when applicable) and removing the defines and C++ code accordingly. With time, then, all of the define-based config initialization code can go away, and we can stop doing per-board builds in GitHub. This doesn't interfere with initUnset... until then, as whatever is not specified gets initialized normally.

We won't need a .h file to understand board defaults anymore, because it'll be there, self-documented in the binary and viewable via gp2040ce-binary-tools. This should help diagnosis and maintenance, and hopefully we can get some vendors on board with this methodology (once all the porting is done) and make it easier for them to configure and ship properly-configured binaries without having to maintain code.

Worth noting that the board config is read by the firmware, but not modifyable, meaning the user config can be cleared to return to board defaults, same as before.

Again this is just step one of the process, but this is the step that lets us start strangling off the BoardConfig.h way of doing things.

NOTE: support for legacy (pre-protobuf) configs had to be removed because of competing expectations of what it meant for loading the config to "fail".

bsstephan avatar Apr 14 '24 05:04 bsstephan

Marked this as a draft because I still have some testing to do, and because I also first want to see how the board builds go in GitHub.

bsstephan avatar Apr 14 '24 05:04 bsstephan

Short summary of what this does, to encapsulate the details above:

  1. Removes the legacy pre-protobuf config code.
  2. Introduces JSON representations of the protobuf config as board configs --- boardVersion is meaningless to the running code and gets replaced in the user config, so this is just for a proof of concept so far.
  3. Patches the protobuf board config into the UF2 file created by the build.
  4. Changes the config loading at boot to: first load protobuf defaults, then load the board config from flash (written by the UF2), then load the user config from flash (which works as it always has)
    1. The old loader, as an aside, would try to load the user config, or load the protobuf defaults if that failed. That was a fallback process, this is more of an overlay process.
  5. Call initUnset... to fill in anything not handled by the board or user configs. As before this overrides defaults and leads to per-BoardConfig.h settings getting saved to the user config.

This does not:

  • Move away from UF2s; they are still built and the primary distributable.
  • Replace BoardConfig.h; that may happen in the future, if and only if board-config.json can cover everything we need.
  • Change the upgrade behavior; initUnset... still functions as before.

It does:

  • Introduce board-config.json in the middle of the config builder; so we can use protobuf features/definitions rather than having to also create duplicative C define directives.
  • Start defining more default values in config.proto; so that board configs only have to provide deltas from the norm, and furthering the amount of C defines we don't need anymore.
  • The above does mean we can start whittling down BoardConfig.h files as we find things they don't need to specify because they are now done via native protobuf.

bsstephan avatar Apr 19 '24 16:04 bsstephan

Future PR will include examples of what it looks like to whittle down the configs, as things are removed or replaced with protobuf representations, and will possibly/probably also include a tool or script or something to convert a whole .h to .json.

bsstephan avatar Apr 19 '24 16:04 bsstephan

Just a further note for anyone else reading this (including myself!), this is a step-us-in-the-right-direction PR change.

This is what is leading us down the road of using .json as configs, and then the UF2 generation will be somewhat seamless. So our builds right now are this:

  1. Include the boardconfig.h for the given "board name"
  2. Rebuild the entire C++ SDK
  3. Generate the uf2 from this setup

This will change us to:

  1. Build the entire C++ SDK
  2. Generate the uf2
  3. Transfer the .json configs into the uf2

Our build times will go down significantly and it will be much easier to build a custom config uf2 for non-technical users.

Going to wait on other folks before I merge this one, but code looks good!

arntsonl avatar Apr 19 '24 16:04 arntsonl

This will be a 0.8.0 feature as we will completely remove legacy config at this point.

arntsonl avatar Apr 22 '24 19:04 arntsonl

Moving this to draft as we have a lot of things to do for protobuf

arntsonl avatar May 03 '24 14:05 arntsonl