Sys Runtime Configuration Registry

[LasseRosenow]

RIOT Runtime Configuration Registry

This is a continuation of the effort previously done in #10622, but its architecture has dramatically changed since then.

Abstract

This PR implements a system level runtime configuration system for RIOT.

A runtime configuration system is in charge of providing a mechanism to set and get the values of Configuration Parameters that are used during the execution of the firmware, as well as a way to persist these values. Runtime configurations are deployment-specific and can be changed on a per node basis. Appropriate management tools could also enable the configuration of nodes.

Examples of runtime configurations are:

• Transmission duty cycles
• Sensor thresholds
• Security credentials
• System state variables
• RGB-LED colors

These parameters might have constraints, like a specific order to be applied (due to interdependencies) or value boundaries.

The main advantages of having such a system are:

• Easy to apply per-node configuration during deployment
• No need to implement a special mechanism for per-node configurations during firmware updates (only in the case of migration), as the parameters persist.
• Common interface for modules to expose their runtime Configuration Parameters and handle them
• Common interface for storing Configuration Parameters in non-volatile storage devices

Modules

The RIOT Registry consists of various modules that are not all needed for minimal operation:

registry
Core functionality such as set, get, export and apply operations to retrieve and set configurations.

registry_storage
Adds capability to safe parameters to non-volatile storage devices.

registry_int_path
Allows accessing configuration parameters via a path of integers.

registry_string_path
Allows accessing configuration parameters via a path of strings.

registry_cli
Allows accessing configuration parameters via a shell command.

Design

Architecture

The proposed architecture, as shown below, is formed by one or more Applications or Configuration Managers and the RIOT Registry. The RIOT Registry acts as a common interface to access Runtime Configurations and store them in non-volatile storage. All runtime configurations can be accessed either from the RIOT application using the provided RIOT Registry interfaces or through the interfaces exposed by the Configuration Managers. A RIOT Application may interact with a Configuration Manager in order to modify access control rules or enable different exposed interfaces.

Path Based Configuration Managers (Needs int_path or string_path extension)

These Configuration Managers are a simple representation of the default configuration structure of the RIOT Registry. They use either the int_path or the string_path Registry extension module to expose the parameters using a path.

Custom Schema Based Configuration Managers

These Configuration Managers have their own configuration structure (custom predefined object models etc.) and can not automatically be mapped to / from the RIOT Registry itself. To make them work, a custom mapping module needs to be implemented, which maps each Configuration Parameter from the registry to the correct format of the Configuration Manager.

Namespaces and Storages

The RIOT Registry interacts with RIOT modules via Configuration Schemas, and with non-volatile storages via Storages. This way the functionality of the RIOT Registry is independent of the functionality of a module or storage implementation. It is possible to get or set the values of Configuration Parameters. It is also possible to transactionally apply configurations or export their values to a buffer or print them. To persist Configuration Values, it is possible to store them in non-volatile storages.

Any mechanism of security (access control, encryption of configurations) is not directly in the scope of the Registry but in the Configuration Managers and the specific implementations of the Configuration Schemas and Storages.

The graphic below shows an example of two Configuration Namespaces (SYS and APP). The APP namespace contains a application specific My app Configuration Schema and the SYS namespace specifies a WLAN and a LED Strip Configuration Schema. The application My app uses the custom My app Configuration Schema to expose custom Configuration Parameters to the RIOT Registry and the drivers WS2812, SK6812 and UCS1903 contain instances of the LED Strip Configuration Schema to expose common LED Strip Configuration Parameters.

Also, there are two Storages available: MTD and VFS. The MTD Storage internally uses the RIOT MTD driver and the VFS Storage internally uses the RIOT VFS module.

Components

The RIOT Registry is split into multiple components as can be seen in the graphic below:

Registry Core

This component holds the most basic functionality of the RIOT Registry. It allows to set and get Configuration Values, transactionally apply them to make the changes come into effect and export all Configuration Parameters that exist in a given Configuration Namespace, Configuration Schema or Configuration Group.

Furthermore it is possible to add Configuration Namespaces or Configuration Schema Instances.

Registry Namespace

The Configuration Namespaces such as SYS or APP and their respective Configuration Schemas are not part of the Registry itself. It is possible to add custom Configuration Namespaces depending on the given needs.

Storage

The Storage component provides an interface to load Configuration Values from a persistent Storage implementation or to save the current Registry configuration to it.

Registry Storage [Extension]

The implementations of a Storage such as VFS or MTD are not part of the Registry itself and can be switched out with implementations that are most suitable to the given needs.

Integer Path [Extension]

The int_path component provides helper functions that convert a path of up to 4 integer values to the respective pointer of a Configuration Namespace, Configuration Schema, Configuration Schema Instance, Configuration Group or Configuration Parameter and the other way around.

The structure of an integer configuration path is the following:
namespace_id / schema_id / instance_id / group_id | parameter_id

For example:

Path	Result
0	Namespace Object
0 / 1	Namespace and Schema Object
0 / 1 / 0	Namespace, Schema and Instance Object
0 / 1 / 0 / 3	Namespace, Schema, Instance and Group or Parameter Object

String Path [Extension]

The string_path component provies helper functions that convert a string path to the respective pointer of a Configuration Namespace, Configuration Schema, Configuration Schema Instance, Configuration Group or Configuration Parameter and the other way around.

The structure of a string configuration path is the following:
namespace_name / schema_name / instance_id (/ group_name)* / parameter_name.

The amount of path items is flexible, so the path could only consist of the namespace_name or only the namespace_name and the schema_name and so on.

For example:\

Path	Result
sys	Namespace Object
sys / temperature_pressure_humidity	Namespace and Schema Object
sys / temperature_pressure_humidity / 0	Namespace, Schema and Instance Object
sys / temperature_pressure_humidity / 0 / calibration	Namespace, Schema, Instance and Group Object
sys / temperature_pressure_humidity / 0 / calibration / humidity	Namespace, Schema, Instance, Group and Parameter Object
sys / temperature_pressure_humidity / 0 / last_reading_timestamp	Namespace, Schema, Instance and Parameter Object

API

The graphic below shows the API of the RIOT Registry. The top shows the Core API to manage Configuration Parameters. On the right-hand side are functions to set and get Configuration Parameters, transactionally apply them and export them to a buffer or terminal. On the left-hand side are setup functions to add Configuration Namespaces and Configuration Schema Instances to the Registry.

The bottom shows the storage API to manage the persistance of Configuration Parameters. The left-hand side shows functions to load” and save Configuration Parameters to and from the persistent Storage. The right-hand side shows functions to add Storage Sources (for reading) and to set a Storage Destination (for writing). The Registry can have multiple Storage Sources, but always only one Storage Destination. This allows to migrate from an old Storage to a new one.

The functionality of these functions is explained in the following paragraphs.

Core API

Get

A Configuration Value can be retrieved using the registry_get function. The function takes the registry_node_t and a registry_value_t pointer (to return the value) as its arguments.

int registry_get(
    const registry_node_t *node,
    registry_value_t *value
);

Set

A Configuration Value can be set using the registry_set function. The function takes the registry_node_t, a void* buffer and the buffer size as its arguments.

The buffer must contain the value in its correct c-type. If the Registry expects a u8, but a u16 is provided, the operation will fail. Furthermore the registry can specify constraints like minimum and maximum values and an array of allowed or forbidden values. If these constraints are not fulfilled, then the operation will fail as well.

int registry_set(
    const registry_node_t *node,
    const void *buf,
    const size_t buf_len,
);

Apply

Once the value(s) of one or multiple Configuration Parameter(s) are changed by the registry_set function, they still need to be applied, so that the new values are taken into effect.

Configuration Parameter(s) can be applied using the registry_apply function. In this case the Registry provides multiple apply functions to allow applying in varying degrees. The provided functions are registry_apply, this function applies every Configuration Parameter currently available in the Registry within the specified scope, specified by the registry_node_t argument.

When a whole Schema Instance, or a single Configuration Parameter is applied, it will be passed on to the apply_cb handler of the Configuration Schema Instance, provided by the module that needs runtime configuration. This way the module gets notified, when the Configuration Parameter has been applied and can apply the changes accordingly.

int registry_apply(const registry_node_t *node);

Export

Some times it is convenient to have a way to see what Configuration Namespaces, Configuration Schemas, Configuration Schema Instances, Configuration Groups or Configuration Parameters are available within our current RIOT Registry deployment. To get this information there is the registry_export function.

This function exports every Configuration Object currently available in the Registry within the scope of the provided registry_node_t argument.

When a node in the schema is exported, it will be passed on to the export_cb handler provided as an argument of each registry_export* function.

int registry_export(
    const registry_node_t *node,
    const registry_export_cb_t export_cb,
    const uint8_t tree_traversal_depth,
    const void *context
);

Add Namespaces to the Registry

To be able to use Configuration Schemas and their Parameters etc. it is necessary to add a Configuration Namespace that holds the required Configuration Schemas to the Registry.

This is possible using the REGISTRY_ADD_NAMESPACE macro, providing the name of the Configuration Namespace and a pointer to a registry_namespace_t object as arguments.

#define REGISTRY_ADD_NAMESPACE(_name, _namespace)

Add Configuration Schema Instances to the Registry

To implement runtime configuration functionality into a module, it is necessary to add a Configuration Schema Instance of the needed Configuration Schema to the Registry.

This is possible using the registry_add_schema_instance function, providing the Configuration Schema and the Configuration Schema Instance as arguments.

int registry_add_schema_instance(
    const registry_schema_t *schema,
    const registry_instance_t *instance,
);

Storage API (Extension)

Load from Storage

It is often needed to load Configuration Parameters from a non-volatile Storage device. For example when a device restarts after a shutdown.

This is possible using the registry_storage_load function. This function takes a registry_storage_instance_t as its arguments. The the registry_storage_instance_t contains data such as the mount point. Internally the Storage Instance searches its persistent storage device for Configuration Values. If a Configuration Value is found, the Storage Source calls the registry_set function and provides the necessary arguments.

int registry_storage_load(
    const registry_storage_instance_t *storage_instance,
);

Save to Storage

To save Configuration Values to a non-volatile Storage device, the Registry provides the registry_storage_save function. This function saves all available Configuration Parameters within the scope specified by the registry_storage_node_t argument.

Internally these functions call the save handler of the provided Storage Instance for each Configuration Parameter that has to be saved to storage. The save handler of the Storage, of the provided Storage Instance takes a Storage Instance providing data such as the mount point, a registry_node_t, which describes the location inside the configuration tree and a Configuration Value as its arguments. These values provide the needed information to load them back into the registry.

The way how the Storage stores these values internally is not specified.

int registry_storage_save(
    const registry_storage_instance_t *storage_instance,
    const registry_node_t *node,
);

Set Storage Instances

To be able to expose storage instances to configuration managers such as the RIOT CLI, we have a function called registry_storage_set_instances, which lets you provide an array of storage instances.

registry_error_t registry_storage_set_instances(
    const registry_storage_instance_t **storage_instances,
);

Get Storage Destination

To be able to retrieve a list of existing storage instances, we have a function called registry_storage_get_instances, which takes a pointer to an array of storage_instance_t structs as an output parameter.

registry_error_t registry_storage_get_instances(
    const registry_storage_instance_t ***storage_instances,
);

Comparison to Apache Mynewt Config

While originally our work on the RIOT Registry was heavily inspired by Apache Mynewt Config, it has since evolved to provide features such as Type Safety, make String Paths optional, introduce Integer Paths and provide a more modular Pointer-Based API.

The table below shows the difference between Apache Mynewt Config and the proposed RIOT Registry.

The Idea here is not to talk down Apache Mynewt Config, as its simplicity of course has its own advantages, but to point out more clearly how our solution differs to it.

Feature	Mynewt Config	RIOT Registry
Pointer based API	❌	✅
Integer Path based API	❌	✅
String Path based API	✅	✅
Shared Configuration Schemas with instances	❌	✅
Nested configuration groups / parameters	✅	✅
Parameter types (string, int8, uint32, float, ...)	❌	✅
Internal parameter value format	String	Any (defined by schema)
Persistent configuration	✅	✅
Applying multiple changes at once	✅	✅

External Configuration Managers

CLI

The only available Configuration Manager in this PR is a CLI.
It can be tried in the example under examples/registry.

The CLI uses int paths separated by /.
E.g.: 0/0/0/0.

CoAP API

The idea is to use the registry_int and/or registry_path module to provide a simple CoAP API.

/get endpoint

The get function could be mapped to the CoAP GET function.

/set endpoint

The set function could be mapped to the CoAP PUT function.

/apply endpoint

This is more tricky. Possible solutions could be to provide an /apply suffix or prefix at the end or beginning of a path, to tell the registry, that this is an apply operation.

/export endpoint

The export function could be implemented using a CoAP GET function using /export as a suffix or prefix of the path. CoAP would then return one response containing all the requested objects for example structured using CBOR.

LwM2M

A LwM2M integration would require to write a mapping between each LwM2M Object to the respective RIOT Registry Configuration Schemas.

In this case every LwM2M set operation would immediately trigger a registry_apply as LwM2M does not provide an apply operation. This is not a drawback, as LwM2M allows to set multiple values at the same time, thus covering the same use-case as the registry_apply function.

Testing procedure

Testing commands unter tests/unittests:

make tests-registry term

make tests-registry_storage term

make tests-registry_storage_heap term

make tests-registry_storage_vfs term

make tests-registry_int_path term

make tests-registry_string_path term

Issues/PRs references

See also #10622 See also #19557

[waehlisch]

thanks for this very careful PR description, including the feature matrix comparison!

[LasseRosenow]

Based on the metrics only, I doubt this has a chance to get merged. The amount of lines of codes IMO is way too large to justify adding it.

Thanks for your reply, conveying your concerns regarding the size of this PR.

I want to do some clarification regarding the actual code size of this PR. -- In-Fact most of my code lies in tests and also the core module itself is only 1.668 lines long and also includes a bunch of ìfdef`s that would usually not make their way into the compiled code. These macros make the code quite a lot larger, but they were necessary to add checks regarding minimum maximum and allowed values.

We could discuss removing this feature. It should default to be turned off at the moment anyways since I would argue, that very constrained devices don't want to waste processing time on these checks.

Okay to give more insight here is a list of the different modules of my implementation and their sizes:

Core + Extension modules

Module	Size in lines of code	Explanation
Registry Core	1,668	Basic functionality (already works good enough for LwM2M mappings
Persistent Storage Extension	406	Infrastructure to allow to write to Storages
Int-Path Extension	695	Totally optional (Allows to find Schemas using an integer path
String-Path Extension	943	Totally optional (Allows to find Schemas using a string path

Configuration Schemas (Code-Generated)

Configuration Schemas are generated by a python script and usually only the schema that is used will be compiled.

Module	Size in lines of code	Explanation
Sys-Namespace	539	Contains a basic RGB LED Configuration Schema
Tests-Namespace	3,146	Contains many large configuration Schemas for various testing scenarios

Storage Implementations

Module	Size in lines of code	Explanation
Heap-Storage	119	Example storage implementation using heap memory
VFS-Storage	349	Example VFS based storage implementation

Applications

Module	Size in lines of code	Explanation
Example Application	158	Shows how to use the Registry API
CLI Application	501	Registers a CLI to the RIOT Shell as an interface to the Registry

Tests

Module	Size in lines of code
Core Test	1,987
Persistent Storage Test	226
Int-Path Test	315
String-Path Test	285
Heap-Storage Test	156
Storage-VFS Test	183

[OlegHahm]

From my perspective the lines of code are much less relevant compared to the added ROM requirements. Well, yes, this is an optional feature, but it still doesn't do any good when people can't use it in the majority of the cases because ROM would overflow.

I only did a very quick test comparing the provided registry example application with a similar application without it: the difference (on nrf52) was ~60k vs ~22k of ROM - that's indeed huge. However, of course this comparison is not quite fair and very superficial. Do you have more elaborated data on how big the actual overhead compared to alternate solutions would be?

[LasseRosenow]

I will do more detailed measurements on the ROM overhead in the coming weeks.

It should be considered that the current example also has a dependency on lifflefs, VFS etc. Which is not necessarily a requirement for the persistent storage.

So more insight is necessary when comparing the ROM overhead of the different parts of the registry.

[benpicco]

Hey, sorry for the reactions being so negative. The problem with the config system is that everybody wants to have a config system for their subsystem, we can expect that a config system will be part of most firmwares in the future, so it's overhead can not be too big.

I understand that this PR includes all possible and optional features, but to better reason about the solution, can you provide a minimal viable config system core that just includes the basic functionality (without any optional dependencies).

We should also understand that we might to reach the perfect solution on the fist try, but if the API is sane we can always improve the implementation later on.

[LasseRosenow]

Hey, sorry for the reactions being so negative. The problem with the config system is that everybody wants to have a config system for their subsystem, we can expect that a config system will be part of most firmwares in the future, so it's overhead can not be too big.

I understand that this PR includes all possible and optional features, but to better reason about the solution, can you provide a minimal viable config system core that just includes the basic functionality (without any optional dependencies).

We should also understand that we might to reach the perfect solution on the fist try, but if the API is sane we can always improve the implementation later on.

Yeah no worries that is already on my list for early January, sorry that I haven't really found time to provide the more detailed measurements yet, it became a bit more busy for me towards the end of the year and Christmas :)

I think there is a need for me to provide better explanations of what each extension is used for and what their costs are.

I was planning to make a small user story for a minimal registry that for example gives some basic ability to manage the on board LED.

Anyways thank you for your feedback, it is very much appreciated by me :)

[benpicco]

Alternatively you could also see if #19557 covers your use case and if not, what's lacking - @fabian18 is getting paid to work on this and will likely maintain it for the years to come, so without having looked into either PR extensively, that would me more inclined towards his solution. On the other hand, having less work on our side is always welcome too, so if you think your solution is better, go ahead with it.

[waehlisch]

Alternatively you could also see if #19557 covers your use case and if not, what's lacking

to not lose track of ongoing discussions elsewhere: https://forum.riot-os.org/t/runtime-configuration-breakout-session/4006

[LasseRosenow]

So I have added a minimal example application under examples/registry_core, which continuously toggles the BOARD LED on and off.

As this is the minimal example, it should be noted that it does NOT include a Storage Backend. I believe that this is a reasonable thing to do, because the backend API of Fabian and me is very similar and in the end we could both just merge each others Storage Backend implementation.

Running make BOARD=same54-xpro cosy on it shows the following results:

(I have chosen the same BOARD as Fabian for better comparison)

What can this minimal example do?

The minimal example includes enough capabilities of the RIOT Registry to integrate well with External Configuration Managers such as LwM2M or OPC UA. (These integrations are not implemented). These External Configuration Managers have their own specified Configuration Schemas, onto which the RIOT Registry can map using its static pointer based API.

What can't this minimal example do?

The minimal example can not integrate with dynamic External Configuration Managers that need String Paths or Integer Paths, such a Custom CoAP API.