sys: runtime configuration module with persistent storage backends

[fabian18]

Contribution description

This provides a generic configuration module for setting/getting runtime configuration parameters of modules which require to store configuration parameters on persistent storage. It should serve the same purpose as Zephyres settings subsystem. It is maybe a bit more limited because the zephyre module seems way more complex to me. Multiple backend can be implemented in the future. The first backend implementation is included and uses the FlashDB. Read the documentation in configuration.h for more details.

New modules:

• configuration: base module
• configuration_backend_flashdb_vfs: enable FlashDB VFS mode as a configuration backend
• configuration_backend_flashdb_mtd: enable FlashDB FAL mode as a configuration backend
• configuration_backend_reset_flashdb: erase configuration from FlashDB backend
• auto_init_configuration: 2nd level of auto-init XFA calling all auto-init functions of configuration subsystems
• auto_init_configuration_backend_flashdb: auto-init FlashDB backend

Testing procedure

There is a test in tests/configuration. By default it uses a pseudo backend configuration_backend_ram.

$ make

main(): This is RIOT! (Version: 2023.04-devel-873-gc4750-pr/runtime_configuration)
.......
OK (7 tests)
Tests done

You can also try the test with FlashDB:

$ TEST_CONFIGURATION_BACKEND=configuration_backend_flashdb_mtd make

[I/FAL] Flash Abstraction Layer (V0.5.99) initialize success.
main(): This is RIOT! (Version: 2023.04-devel-873-gc4750-pr/runtime_configuration)
.....
OK (5 tests)
Tests done

When switching from FlashDB MTD to FlashDB VFS a format is required. $ USEMODULE+=vfs_auto_format TEST_CONFIGURATION_BACKEND=configuration_backend_flashdb_vfs make

main(): This is RIOT! (Version: 2023.04-devel-873-gc4750-pr/runtime_configuration)
.....
OK (5 tests)
Tests done

Issues/PRs references

It can help to solve #16844

[fabian18]

@DanielLockau-MLPA FYI

[chrysn]

I'd be glad to have a config system in the style of this; before I jump into code, but do have questions:

• Are there use cases that warrant the complexity of having different back-ends "mounted" at different paths?
• Is an intention of this to store data across firmware updates? If so, how should keys be versioned?
• I understand this to not have any typing; all data items are arbitrarily long byte sequences. Should there be helpers for the typical case of accessing a u32 or a bool?
• flashdb on VFS is one of the provided impls. Does that make sense? It's building a key-value store on a database on a key-value store. A plain VFS backend would make much more sense to me.
• As Ben said, the caching / committing model could be addressed in the overview documentation.
• I'd appreciate a note on whose responsibility it is to fail safe in case of power interruptions. (From my current understanding, that needs to be the responsibility of the driver's export function, possibly in combination with the import).
• This is doing a lot of work with text strings. In the test they're rather short, but at one point one may want to avoid collisions, and we might wind up in /org.riot-os.sys.ieee802154/configured-networks/1/rfc9031key space of name sizes. What considerations led to using strings as opposed to, say, enums? (also related to the versioning question above)
• Hah, that prompted a follow-up: Storing network configurations or keys may easily send us into array-of-structs land. Is encoding child names as ASCII numbers really the way to go here?
• If information about which keys is available at build time (eg. in the enum case), could storage backends benefits from knowing the corresponding types? Or would that needlessly increase complexity? It seems that the test does just that, but hardcoded (it just knows which keys will be used). Is that a back-end that's aligned with the system's design goals? If yes, how is the developer supposed to keep the two parts in sync?
• In my mental model of configuration storage on a microprocessor, a double-buffered pair of flash pages containing a struct is the simplest case. Would that work with this model outside a test?

---

(I'm setting this off visibly because while I hope the above can all be answered easily, this might be a rabbit hole you may prefer not to go into:)

• Have you looked at how YANG (eg. in RESTCONF, the full YANG ecosystem is huge) models things? While YANG is not designed as a storage but as an exchange format, it could have some valuable input, especially if the next task is to do remote configuration into this store.

[LasseRosenow]

Hello everyone, I just saw this pull request and would like to add my 2 cents. Over the past year I have also been working on this problem myself as part of my Bachelor Thesis at HAW Hamburg. I am currently working on finishing up my branch to open a PR for this as well :) So I think it might be worth sharing some basic points of my approach here beforehand, as it addresses many of the points that chrysn mentioned in his post.

Also I think maybe this is a good opportunity to start a collaboration between the author of this PR and myself, if interest exists, as we both have put a lot of thinking in solving this issue of runtime configuration in RIOT OS.

I also started to build my architecture the same way as the Zephyr Settings module, based on the prior work here: #10622 But in the end created something totally different. I will list the main differences below, without going too much into detail. A more advanced description of my architecture will follow in the coming weeks when I finally open my PR.

Handler => Configuration Schema

• I decided that having drivers of the same kind implement handlers that will have the same structure is unnecessary duplication of work
• So I decided to have a Configuration Schema that contains the basic implementation, structure and all necessary Configuration Parameters
• A driver then only needs to implement Instances of this Configuration Schema and register them on the Configuration System
• RIOT would then specify basic Configuration Schemas in the SYS namespace, which then are then used by all the drivers/modules
• An application that uses RIOT can also specify its own Configuration Schemas in their own namespace

Fully typed

• The Configuration Schema has type definition for every configuration parameters.

String path => Integer path

• I decided to opt for a struct containing multiple uint_32_t properties to represent the Path that identifies a Configuration Parameter instead of using strings
  • namespace_id (e.g. SYS, or APP)
  • schema_id (e.g. RGB-LED Schema)
  • instance_id (A Configuration Schema can be implemented by multiple drivers as instances)
  • resource_id (Configuration Parameter or Configuration GroupID inside of theConfiguration Schema`)

Move the whole Path into its own module

• I found for simple use-cases that only want to use the storage to persist values, it is not necessary to have a Path as a key to access Parameter values.
• I think a Path based API is mostly for the use-case of Remote Configuration Management etc.

2 kinds of storage backend interfaces

• Pointer based interface
  • Uses the pointer as the key and a tuple of (size, value) as the value
  • There is an example VFS and a example MTD implementation for this backend
• Path based interface
  • Uses the Integer Path as the key and a tuple of (size, value) as the value
  • There is an example VFS and a example MTD implementation for this backend

[fabian18]

@chrysn I have spent some time to reason about and answer your questions.

Are there use cases that warrant the complexity of having different back-ends "mounted" at different paths?

"Different back-ends mounted at different paths", so for example /my/large/config/foo I would put maybe on the FlashDB which should not be exported so frequently but the frame counter which increments for sending an IEEE.802.15.4 frame I would store on a location which is fast to read from and write to. So I think there are use cases where you want to have different backends. Or phrased differently, the assumption that the whole system configuration has to be stored on one backend is a limiting factor.

Is an intention of this to store data across firmware updates? If so, how should keys be versioned?

Firmware 1.0.0 could for example seek for "conf/FW1.0.0/objects" and Firmware 2.0.0 "conf/FW2.0.0/objects". I would assume the firmware knows its firmware version and is able to construct the right key.

I understand this to not have any typing; all data items are arbitrarily long byte sequences. Should there be helpers for the typical case of accessing a u32 or a bool?

For the backend it is just bytes. For the internal representation in RAM it is a struct which is parsed from the backend bytes. So for example an import handler could read CBOR bytes from the backend and parse it to an internal struct. Export would do the opposite.

flashdb on VFS is one of the provided impls. Does that make sense? It's building a key-value store on a database on a key-value store. A plain VFS backend would make much more sense to me.

We could drop the FlashDB VFS backend if you want. It was good for me to see that it works. However it is notably slower than the FAL backend.

As Ben said, the caching / committing model could be addressed in the overview documentation.

Ok I can improve the documentation. I would call the internal struct a cache which is modified with get/set. And when desired can be exported to storage.

I'd appreciate a note on whose responsibility it is to fail safe in case of power interruptions. (From my current understanding, that needs to be the responsibility of the driver's export function, possibly in combination with the import).

Thats a heavy task. I would pass the problem to the backend :D I mean FlashDB advertises with this Support Power-off protection function, high reliability;. And I remember that littlefs2 does so too.

This is doing a lot of work with text strings. In the test they're rather short, but at one point one may want to avoid collisions, and we might wind up in /org.riot-os.sys.ieee802154/configured-networks/1/rfc9031key space of name sizes. What considerations led to using strings as opposed to, say, enums? (also related to the versioning question above)

I agree the string code is ugly but could be good if it comes to URIs for remote confiuration. Strings of course are the first natural idea I think. With enums I would not know how to build a tree. So it would be one enum per handler, I guess? I want that code outside of RIOT can easyly hook up to the configuration handler data structure (the tree). If I had to do it with enums I would create an XFA of handlers which store their enum value and the XFA must be iterated over to find the handler. I think strings are less likely to collide than enums. There should be a way to address all sub-enums with one enum like ther is the way to select a whole subtree. You could give input to an enum configuration subsystem. And I remember that the discussion also happened for the zephyre implementation but I cannot find it anymore. I feel that strings are more flexible and easier to handle/understand than enums.

Hah, that prompted a follow-up: Storing network configurations or keys may easily send us into array-of-structs land. Is encoding child names as ASCII numbers really the way to go here?

I did a configuration array of WiFi access points with it, and yes it resulted in an enumeration /wifi/ap/0 .../wifi/ap/1. The configuration handler was at /wifi/ap and it handled a remainnig integer in the key or if there was no integer, the whole array. The backend key also was /wifi/ap/x. The firmware has allocated an array of access points and tried to import /wifi/ap/x for each array index x. The backend key /wifi/ap would maybe also be possible but the backend configuration system should refuse to import the whole /wifi/ap from the storage when there are unsynced modifications, that means when the configuration is "dirty" so to say. The Wifi configuration has an API build around the configuration API. The Wifi configuration API deals with SSID strings but searches in the WiFI access point array the index of an element with that SSID.

If information about which keys is available at build time (eg. in the enum case), could storage backends benefits from knowing the corresponding types? Or would that needlessly increase complexity? It seems that the test does just that, but hardcoded (it just knows which keys will be used). Is that a back-end that's aligned with the system's design goals? If yes, how is the developer supposed to keep the two parts in sync?

That the backend knows the struct type or at least the maximum size of a configuration item would ease implementation of it. That the backend knows all the keys was just a simple test implementation. In a dynamic approach there would be some reserved memory to store meta information like keys and their offset in memory. In the declaration of conf_backend_load_handler and conf_backend_store_handler there is a size parameter. In the declaration of conf_data_export_handler and conf_data_import_handler I did not include the size parameter because those are implemented together with or in the same file likely, as the internal configuration struct.

In my mental model of configuration storage on a microprocessor, a double-buffered pair of flash pages containing a struct is the simplest case. Would that work with this model outside a test?

Let's say there are configuration structs A, B and C and they fit on a flash page. There would be a handler and an internal allocated instance for each of struct A B and C. The get() and set() would modify the internal instance and export() would call store() to sync the structs to the flash page. Maybe on another flashpage, the backend would need to store meta information like on which offset is which key stored, and add such an entry for every key that is added. This PR is not intended to facilitate backend implementation. It is expected that the backend is capable to allocate memory for new keys and values. The test backend is static though.

[chrysn]

On Mon, May 15, 2023 at 08:22:03AM -0700, fabian18 wrote:

"Different back-ends mounted at different paths", so for example /my/large/config/foo I would put maybe on the FlashDB which should not be exported so frequently but the frame counter which increments for sending an IEEE.802.15.4 frame I would store on a location which is fast to read from and write to.

I agree there are different desirable characteristics, but the frame counter example shows that the mount point paradigm may not be ideal.

Take for example the state of an OSCORE security context. Its key is immutable, the sequence number is changed often (and committed to flash every 256-or-so messages), and the replay window is only ever committed at shutdown (to some value) and at startup (before processing the first message, to an "uninitialized" value).

Managing them at three points in the tree (say, /frozen/oscore/ctx/0/key, /once-in-a-while/oscore/ctx/0/ssn, /at-boot/oscore/ctx/0/replay sounds hard to manage consistently.

Firmware 1.0.0 could for example seek for "conf/FW1.0.0/objects" and Firmware 2.0.0 "conf/FW2.0.0/objects". I would assume the firmware knows its firmware version and is able to construct the right key.

I was rather thinking in terms of smaller changes -- a 2.0 upgrade would likely convert and erase the old config. But an incremental upgrade might change the interpretation of some key, or the permitted range. Would a version 1.1 that replaces the /foo/replay scalar with a struct remove the replay key and write a /foo/replay2 key in the same transaction?

I understand this to not have any typing; all data items are arbitrarily long byte sequences. Should there be helpers for the typical case of accessing a u32 or a bool?

For the backend it is just bytes. For the internal representation in RAM it is a struct which is parsed from the backend bytes.

That conversion sounds like a task that could gain efficiency from some common tools -- but maybe that's better deferred to later in the thread.

I'd appreciate a note on whose responsibility it is to fail safe in case of power interruptions.

Thats a heavy task. I would pass the problem to the backend :D

That's totally fine! I wasn't expecting any mechanism here, just a clear statement what the backend needs to provide.

I agree the string code is ugly but could be good if it comes to URIs for remote confiuration.

At least they come from CORECONF URIs, then these URIs are just encodings for numeric keys (if at all; in the latest revision it'll mainly be CBOR in FETCH payloads IIUC).

With enums I would not know how to build a tree.

I don't have a clear suggestion here; could be that there are enums for the individual path components (say, mapping every string between slashes to an int16_t) and it's resolved like that, could be something external like CORECONF's ("huge": 63bit -- still shorter than most strings, and readable quicky as two aligned reads on most platforms) SIDs.

Could just as well be names boiled by some preprocessor system into (driver, struct offset) pairs. Probably depends on the answers to other questions, and on the amount of descriptive pre-processing one is willing to accept. (Like, would it be fine if the mount points / config structure were described in some TOML or YANG file, which then gets preprocesed into a .h file that has all the right macros?).

Is encoding child names as ASCII numbers really the way to go here?

I did a configuration array of WiFi access points with it, and yes it resulted in an enumeration /wifi/ap/0 .../wifi/ap/1.

There are two things that worry me about this approach:

• It's going back and forth between numerics and ASCII strings a lot. (Right now I can't see clearly how often).

• If there should be semantic operations (like, "add a new item with the next number") that need support from the back-end, this is one more point where back-end struct and front-end usage need to be aligned. (Frankly, I'm still not sure whether the idea here is that there would be generic key-value backends, or fixed-structure backends, or an application-specific mix, or a mix that's USEMODULE dependent).

That the backend knows the struct type or at least the maximum size of a configuration item would ease implementation of it. That the backend knows all the keys [...]

What are the expectations users could work with here?

Can a user start implementing their application on a generic back-end, say, flashdb-on-mtd, and switch to a more bespoke back-end when benchmarking indicates the need for it? Will that require changes to the application? Do you expect to see a toolkit for building a more custom back-end for applications that know their data structure?

Let's say there are configuration structs A, B and C and they fit on a flash page. There would be a handler and an internal allocated instance for each of struct A B and C. The get() and set() would modify the internal instance and export() would call store() to sync the structs to the flash page. Maybe on another flashpage, the backend would need to store meta information like on which offset is which key stored, and add such an entry for every key that is added. This PR is not intended to facilitate backend implementation. It is expected that the backend is capable to allocate memory for new keys and values. The test backend is static though.

What determines the granularity of commits (exports?), and how is that communicated?

[fabian18]

Managing them at three points in the tree (say, /frozen/oscore/ctx/0/key, /once-in-a-while/oscore/ctx/0/ssn, /at-boot/oscore/ctx/0/replay sounds hard to manage consistently.

Taking /net/coap/oscore/ctx/0/{key | replay | ssn} I would suppose that the handler sits at /net/coap/oscore/ctx in the tree. The internal data represenation is an array of some struct oscore_ctx_t ctx[NUMOF]. The handler knows the array and the struct layout and can directly access ctx[0].{key | replay | ssn} The backend key under which the backend stores the data could be: 1.) /net/coap/oscore/ctx, so the backend stores the array continuously or

2.) /net/coap/oscore/ctx/x where x is an index.

Maybe to facilitate efficient update, the backend store() function could be passed an offset parameter, so in case 1: export(/net/coap/oscore/ctx, 0/key)is transformed to store(/net/coap/oscore/ctx, &ctx[0], &size, 0 * sizeof(oscore_ctx_t) + offsetof(oscore_ctx_t, key)), where size is sizeof(ctx).

2: export(/net/coap/oscore/ctx, 0/key)is transformed to store(/net/coap/oscore/ctx/0, &ctx[0], &size, offsetof(oscore_ctx_t, key)), where size is sizeof(ctx[0]).

If the backend supports bytewise access it could use the offset parameter for optimized access. If the backend is based on flash and anyways a full page must be written it could just ignore the offset parameter. For load() I would not add an offset parameter, because I would see that the backend would require a temporary buffer to load the full object and only copy the member to the given pointer location to not overwrite the current cache value. The temporary buffer would have to be provided by the import() or dynamically allocated by the backend. I would say this is not worth.

A mount path to a configuration object of which one member must be updated very frequently could store more than one backend pointer in the .c file. With the knowledge about the configuration object, which the handler know about, it should be possible to split an object accross backends. The configuration handlers would implement the selection of the right backend pointer by given key. Let's say there is the file oscore/configuration.c which implements the handlers for the array of oscore_ctx_t.

The export handler could allocate a struct which looks like the oscore_ctx_t, but with the special member removed. This kind of structure is stored by backend_1. Or when just this special member at that offset should be updated the export handler allocates a struct which contains just the member which has been removed from the original struct. And we know backend_2 stores this kind of structure. However the split data that comes from the two backends can be combined in the local array of original oscore_ctx_t. So import() also would have to allocate one structure type or the other and call load() from the right backend and copy the received data to the location in the oscore context struct where it belongs. If a full context should be exported or imported entirely, load and store must be called twice. That means for backend_1 and backend_2.

[fabian18]

It's going back and forth between numerics and ASCII strings a lot.

The concept for a configuration array is that we can know if an index is free or not. A wrapper function to sync an array to a backend, using the configuration API would iterate through the array and delete any unoccupied slots and export occupied ones. For example if configuration array foo[0] is a free slot but 1 is not, the wrapper function should export("/foo/1) and delete("/foo/0"). Besides that you could still export the configuration as a whole with "/foo". It depends on your handler implementation. You can model it as you like.

[chrysn]

Having a TOML file for a subsystem in a /conf/RIOT/202304 folder would be an easy file system backend. The import() function would read the file and parse it into a struct.

I think we're talking different things here -- I meant describing on the PC side in a TOML file the mount points and possibly even the keys in there.

Parsing a TOML file at runtime sounds like it'd exceed the capacities of most RIOT systems. Sure it can be done, at least on the larger ones, but it'd occupy way more flash and stack than I'm comfortable allocating for an embedded configuration system, especially if it is supposed to be used by the OS. (For what it's worth, I think that any text-string processing function on an embedded device indicates a badly designed protocol underneath it, but that's an extremist PoV. But not doing TOML-style complexity on the microprocessor is probably a widely supported approach).

[chrysn]

To better understand the requirements on this, could we compare the proposed interface to the EEPROM "registry"? (Not that I'd be perfectly happy with it either, but to get a few properties straight).

• The EEPROM registry has no structured names, whereas this module uses a slash-delimited structure (but users could just as well name things with slashes in eereg).
• This PR supports different back-ends, and supports having different prefixes assigned to different back-ends.
• This PR supports some kind of staging / prepare-and-commit semantics, but I'm not sure I understand their granularity yet.

Is that about it, or did I miss something?

[fabian18]

Using 64 bit SIDs now as keys. A string path is constructed on handler lookup when configuration_strings is used.

[fabian18]

As of now ...

TEST_CONFIGURATION_BACKEND=configuration_backend_flashdb_mtd USEMODULE+="configuration_backend_reset_flashdb" BOARD=same54-xpro make cosy

Cosy shows about 4K ROM for the module with FlashDB backend.

and 3K ROM without strings BOARD=same54-xpro make cosy

[fabian18]

SID assignment

An SID is a unique identifier for any configuration value. Every instance within an array has its own SID. To integrate a new configuration module, you have to reserve an appropriate SID space for it and assign an SID schema by following some rules.

The global SID space ranges from 0x0000000000000000 to 0xffffffffffffffff. The author of a configurable module must request an unassigned SID subspace. Within the allocated SID space, every nested configuration object should be assigned a sub SID space. Simple configuration values reserve just one SID value. When a configuration type has multiple instances they must be modeled as an array where each instance has its own SID. To assign all instances an SID, you must only assign an SID space for the first element and an SID stride, to model the whole array. The required size depends on the maximum array size which must be known, and the complexity of the array type. For each array, or sub-array you have to assign the SIDs for the first instance, at position 0. The SIDs for every other instance has a constant SID distance to the first item. The SID of an instance can be computed by the base SID of that type and the indices within the array.

SID assignment rules for different types of configuration nodes

Single value node

A simple node which represents one integer, string or byte array value is assigned a single SID within the SID range of its parent node.

Compound value node

A compound object must be assigned an appropriate SID range [sid_lower, sid_upper], which is large enough to include all sub ranges and instances in an array. With a larger reserved SID space a type is able to grow in the future.

Array vale nodes

An array is also just a kind of container type and must be assigned an SID space. The first SID in that range sid_lower targets the entire array for an operation. By convention, sid_lower + 1 targets the array at position 0. The next item in the array can be computed by adding a constant stride parameter stride. The SID range (sid_lower + 1 < X < sid_lower + 1 + stride) is available to assign to members of the array types.

SID assignment example

Given is the schema from the current CORECONF draft:

{
  1723 : "2014-10-26T12:16:31Z" / current-datetime (SID 1723) /
},
{
  1533 : {
     4 : "eth0",              / name (SID 1537) /
     1 : "Ethernet adaptor",  / description (SID 1534) /
     5 : 1880,                / type (SID 1538), identity /
                              / ethernetCsmacd (SID 1880) /
     2 : true,                / enabled (SID 1535) /
    11 : 3             / oper-status (SID 1544), value is testing /
  }
}

Let the SID range 1000 to 1000000000 be the sys configuration range. Let the SID range 1500 to 1699 be the interfaces range. Let the SID range 1532 to 1699 be the array (list) of board network interfaces if[].

That means 1533 could be the first item in the list of interfaces. The SIDs for interface parameters could be: 1534: description, 1535: enabled_status, 1537 :name, 1538: type, 1544: operation_status. There must be a reserved SID space to target the properties of an interface. Let the stride between two interfaces be 30. If the first interface has the SID 1533: sys/interfaces/if/0, the next interfaces would be assigned the SIDs:

1563: sys/interfaces/if/1,
     1564: sys/interfaces/if/1/description
     ...
     1574: sys/interfaces/if/1/operation_status
1593: sys/interfaces/if/2,
1623: sys/interfaces/if/3,
1653: sys/interfaces/if/4,
     1654: sys/interfaces/if/4/description
     ...
     1664: sys/interfaces/if/4/operation_status

Having a at most 5 interfaces is a bit low though. But this is how the SID enumeration of configuration items works in this implementation

Let the SID range 1700 to 3000 be the system-state configuration range. If the SID range 1721 to 1800 is the clock subdomain, 1722: boot_datetime and 1723: current_datetime could be reserved.

In the context of this implementation, node types for all configuration targets must be allocated.

CONF_HANDLER_NODE(sys, 1000, 1000000000): {

    CONF_HANDLER_NODE(interfaces, 1500, 1699): {

        CONF_ARRAY_HANDLER(if, 1532, 1683, 30): [

            CONF_HANDLER(description, 1534): {}

            CONF_HANDLER(enabled_status, 1535): {}

            CONF_HANDLER(name, 1537) {}

            CONF_HANDLER(type, 1538) {}

            CONF_HANDLER(operation_status, 1544) {}
        ]
    }

    CONF_HANDLER_NODE(system_state, 1700, 3000): {

        CONF_HANDLER_NODE(clock, 1721, 1800): {

            CONF_HANDLER(boot_datetime, 1722): {}

            CONF_HANDLER(current_datetime, 1733): {}
        }
    }
}