Configuration management

[mikee47]

Options, options, everywhere!

Sming is highly configurable with a lot of settings. You can see some of them by running make list-config, though that list is actually incomplete (e.g. SPI_* settings aren't in there).

I've opened this issue to discuss the problem in general terms so we can come up with a solid solution and make life easier for both end-users and developers. My proposal is that we adopt KConfig for this, with a graphical menu system to allow editing and adjustment to all relevant settings. Further, settings are fully defined and documented inside each code module which will reduce the need for additional documentation.

ESP-IDF uses a kconfiglib fork to manage configuration data. Unlike the original kconfig it's cross-platform.

Example

At the moment I edit Makefile-Windows.mk to set defaults like this:

SPI_SPEED = 40
SPI_MODE = dio
SPI_SIZE = 4M

COM_SPEED_SERIAL = 74880
COM_SPEED_ESPTOOL = 921600

So they apply globally to any projects. Not an ideal solution, but neither is it a good idea to set them all as global environment variables. In fact, the SPI_* settings above should really be stored as a group, identified by the target hardware (in this case, nodeMCU).

We need a structured way to manage all these settings, both to allow changing of global defaults and on a per-project basis. In addition, the system should accommodate pre-defined lists of configuration values, so we can allow easy selection of target hardware.

We could start by adding make menuconfig, which gets triggered after a fresh install of Sming so the user can set up their required default settings. Also, some settings (like the target hardware) won't have defaults so will trigger menuconfig also.

[aemseemann]

Considering the recent discussion over at https://github.com/SmingHub/Sming/issues/1684, wouldn't cmake-gui be a start?

[mikee47]

Thanks, that is helpful to see, and certainly a lot simpler than adopting a whole separate system for it.

Perhaps here we can discuss how Sming presents these options to the user, rather than how we actually implement them? It's great that CMake has an integrated way to deal with these, but for the time being at least we should be mindful of keeping both build systems broadly in sync.

[mikee47]

For PR #2171 I propose that the information provided by some build variables (see below) is placed instead in a hardware configuration file attached to the project (and under revision control). Each file is specific to a hardware variant. May I propose .hw as the file extension ? For example, widget.hw. We could have widget32.hw for an esp32 version, etc.

This would be supported by a python tool, based on or in addition to gen_esp32part.py. Suggest hwconfig.py ?

Pre-configuration

Instead of defining build variables directly in a project, we add a 'pre-configuration' step which reads widget.hw and generates hwconfig.mk containing the appropriate information. It also generates the binary partition table.

Example:

make configure CONFIG=widget.hw

The build variables are still present, but managed in a more flexible way.

The generator also allows the makefile logic to be simplified, e.g. by performing numeric calculations, hex/decimal conversions, etc. anything which is a pain to do in a makefile.

The .hw file allows different sets of these variables to be generated depending on the hardware being targeted.

Following pre-configuration, project building proceeds as usual. The tool only needs to be run again if any of the information in the .hw file changes: this could be handled automatically by the build system after initial configuration.

Existing projects

Use hwconfig.py to generate widget.hw file from existing project settings. Those settings should then be removed from the project's component.mk file.

Other information widget.hw might contain:

• Build information to create/update the image file for the partition content, e.g. command line, makefile target, etc.
• Description of the target hardware
• Build options relating to that hardware
• Anything which needs to change if switching between different hardware variants

Standard configs

We can provide some standard configs available via the build system. Example:

make create-config SMING_ARCH=Esp8266 TEMPLATE=spiffs CONFIG=widget.hw

We'd then edit the new widget.hw file and then run make configure CONFIG=widget.hw

Affected build variables

• PARTITIONS_CSV

  Taken from esp32. Not required.

  I considered adding HARDWARE_CONFIG to indicate the .hw file to use, however by adding a pre-configuration step this is not necessary.

• PARTITION_TABLE_OFFSET

  Store this in the .hw file.

  This is adjustable to allow more room for the boot sector / config. Everything following the partition table is defined the partition table entries.

  For example, with esp8266:

    0x0000 rBoot boot sector
    0x1000 rBoot config sector
    0x2000 PARTITION_TABLE
    ... entries
  

• DISABLE_SPIFFS

  It does not mean a SPIFFS partition is not available, only that it won't be initialised with data.

  Very confusingly named as it only controls SPIFFS image generation. Not required as partition table provides this information.

• RBOOT_SPIFFS_0, RBOOT_SPIFFS_1, SPIFF_SIZE

  Specified in the partition table.

• SPIFF_FILES, SPIFF_BIN

  Used by SPIFFS image builder. The partition table can contain the appropriate makefile build command to generate the image, and it will use this. Example:

    "spiffs1": {
        "address": "0x200000",
        "size": "0x10000",
        "type": "data",
        "subtype": "spiffs",
        "filename": "$(FW_BASE)/spiff_rom.bin",
        "make": {
            "target": "spiffsgen",
            "content": "files"
        }
    },
  

  The build system can append PARTITION=spiffs1 automatically to the make command so the partition information can be queried (using ~~parttool.py~~ hwconfig.py). For example, our spiffsgen target queries the partition table for content to identify what goes in the image, and filename so it knows where to write it.

• RBOOT_ROM_0, RBOOT_ROM0_ADDR, RBOOT_ROM_1, RBOOT_ROM1_ADDR, RBOOT_ROM2_ADDR

  Goes in the related partition table entry. Pre-configuration will contain appropriate build targets. Example:

    "app_main": {
        "address": "0x8000",
        "size": "0xf8000",
        "type": "app",
        "subtype": "factory",
        "filename": "$(FW_BASE)/rom0.bin",
        "make": {
            "target": "fwimage"
        }
    },
  

• SPI_SIZE

  esptool can (usually) auto-detect flash size so this shouldn't be required. As the critical sectors are now at the start of flash we don't need it for that either. It should go in the .hw file anyway.

[mikee47]

Here is an example to show the general structure of the hardware configuration file. I'll update this as things progress:

{
	"name": "Standard hardware config",
	"arch": "Esp8266",
	"flash-size": "4M",
	"partition-table": {
		"offset": "0x2000",
		"entries": {
			"phy_init": {
				"address": "0x3000",
				"size": "0x1000",
				"type": "data",
				"subtype": "phy",
				"filename": "$(SDK_BASE)/bin/esp_init_data_default.bin"
			},
			"nvs_data": {
				"address": "0x4000",
				"size": "0x4000",
				"type": "data",
				"subtype": "nvs"
			},
			"app_main": {
				"address": "0x8000",
				"size": "0xf8000",
				"type": "app",
				"subtype": "factory",
				"filename": "$(FW_BASE)/rom0.bin",
				"make": {
					"target": "fwimage"
				}
			},
			"app_0": {
				"address": "0x108000",
				"size": "0xf8000",
				"type": "app",
				"subtype": "ota"
			},
			"spiffs1": {
				"address": "0x200000",
				"size": "0x10000",
				"type": "data",
				"subtype": "spiffs",
				"filename": "$(FW_BASE)/spiff_rom.bin",
				"make": {
					"target": "spiffsgen",
					"content": "files"
				}
			},
			"spiffs2": {
				"comment": "Uninitialised reserve SPIFFS partition",
				"address": "0x21000",
				"size": "0x40000",
				"type": "data",
				"subtype": "spiffs"
			}
		}
	}
}

[slaff]

Sming has multiple configuration settings that allow the framework to be tweaked to our needs. At the moment these configuration settings can be found by looking at the documentation or/and at the component.mk files. It would be better to have easier visual selection of those features. For example running in your application a command like the one below

make config

will take care to fetch all configuration settings and present a nice visual dialog.

The above can be implemented:

• using ESP-IDFs fork of python's Kconfiglib library which can be included in Sming.
• adding .config file to every component providing adjustable parameters. The .config file will contain a menu definition with parameters, explanation, default values, etc.
• adding .config to sample applications that provide adjustable parameters
• storing and using the final generated config in $(OUT_BASE)/.config and integrating it with the existing Sming's configuration system.

[mikee47]

As far as hardware configurations/profiles go it would be lovely to have a graphical editor for that. However, rather than being text-based it could present a graphical overview of the memory layout and allow partitions to be dragged around, resized, etc.

Graphical framework for python?

[mikee47]

A limitation of the current hardware configuration/profile layout is that we only have single inheritance. Therefore in situations like this we end up having to create multiple hardware configurations to satisfy one additional use case. This general trend would mean we end up with a confusing array of different configurations with a lot of duplication.

We could express the configuration like this:

{
	"name": "My project configuration",
	"options": ["4m", "spiffs", "vdd"]
}

Note that base_config has been omitted; as a simplification we can default this to standard if not specified.

The list of options can be expanded from a library file like this:

{
    "4m": {
        "devices": {
            "spiFlash": {
                "size": "4M"
            }
        }
    },
    "spiffs": {
        "partitions": {
            "spiffs0": {
                "address": "0x200000",
                "size": "512K",
                "type": "data",
                "subtype": "spiffs",
                "filename": "$(SPIFF_BIN_OUT)",
                "build": {
                    "target": "spiffsgen",
                    "files": "$(SPIFF_FILES)"
                }
            }
        }
    },
    "vdd": {
        "partitions": {
            "phy_init": {
                "filename": "$(FLASH_INIT_DATA_VCC)"
            }
        }
    }
}

Note the 'vdd' fragment which relates to #2264.

Despite the fact a user has to create a custom profile, it might be simpler to comprehend and more flexible than picking from a large list of pre-defined profiles.

We'd just provide the minimal set of standard profiles and leave all the option selection to the user.

[slaff]

Despite the fact a user has to create a custom profile, it might be simpler to comprehend and more flexible than picking from a large list of pre-defined profiles.

I like the idea with the options. If we can generate the hw config for the application without bothering the user it would be even better. How about allowing the hardware config options to be set by makefile configuration and behind the scene generating the final hardware configuration and storing it in the out/Esp8266/... directory?

Example

make HWCONFIG_OPTS=vcc,m4

This will use the current HWCONFIG, for example standard, add to the file content the selected options and generate finally the newly modified $(OUT_BASE)/standard.hw which will be used instead of the original?

[mikee47]

Mmm... that could be baked into standard perhaps... "options": [${HWCONFIG_OPTS}]...

[slaff]

@mikee47 I am trying to include LittleVGL as a library in Sming. LittleVGL is known to have a lot of options that should be configured. And I was wondering if you can extend the build system to have support for KConfig files. This can be based on ESP-IDF's python code.

My rough idea is the following:

• to have a new makefile target called "menu-config" that is checking for Kconfig files in the current application and included libraries, components and core code.
• The KConfig files should be in the root folder of an application, component or library
• Every component can provide default values. For example with a file called "config.defaults" in the root folder of the application, component or library. A default config can be also architecture dependent. For example if there is a "config.Esp8266.defaults" it will be used to override the values in "config.defaults".
• Once a KConfig value is selected it should be stored to the out/<Arch>//config.mk file. If the option name in the KConfig file is CONFIG_something it should be stored in the config.mk file without the CONFIG_ prefix. For example if there is an option called "CONFIG_LV_ENABLE_WIDGETS" it should be stored in the config.mk file as "LV_ENABLE_WIDGETS"

@mikee47 Do you think you can add this to the build system?

[mikee47]

@slaff Curiously I'm working on a graphics library too :-) I'm aiming to get a release in the next week or two so once that's done I'll take a look at this.

[slaff]

I'm aiming to get a release in the next week or two so once that's done I'll take a look at this.

Ou yeeeh!

[slaff]

I'm aiming to get a release in the next week or two ....

@mikee47 what's the progress here ? Do you have something ready that we can test (as WIP PR ) ? If you are right now on an overpriced vacation enjoy and relax as much as possible ;-)

[mikee47]

@slaff Having been messing around with the ESP IDF KConfig system I have a few thoughts about use of KConfig.

Regarding LVGL, yes, there are a lot of settings but from a quick read through there are only a few that the application might want to change, such as LV_USE_PERF_MONITOR. Some settings are associated with the display driver, so wouldn't be configured by the application anyway. Other settings, such as all the LV_FONT_xxx, can all be enabled by the Component since the linker will discard any fonts which aren't used.

* to have a new makefile target called "menu-config" that is checking for Kconfig files in
  the current application and included libraries, components and core code.

* The KConfig files should be in the root folder of an application, component or library

OK

* Every component can provide default values. For example with a file called "config.defaults"
  in the root folder of the application, component or library.
  A default config can be also architecture dependent.
  For example if there is a "config.Esp8266.defaults"
  it will be used to override the values in "config.defaults".

Aren't defaults provided in KConfig? ESP IDF uses the depends on clause to manage this, not sure whether that's an extension or a core part of KConfig.

* Once a KConfig value is selected it should be stored to the out///config.mk file.

OK.

If the option name in the KConfig file is CONFIG_something it should be stored in the config.mk file without the CONFIG_ prefix. For example if there is an option called "CONFIG_LV_ENABLE_WIDGETS" it should be stored in the config.mk file as "LV_ENABLE_WIDGETS"

Why this is necessary?

[slaff]

Why this is necessary?

If I am not mistaken KConfig adds always CONFIG_ prefix to all configuration directives listed in a KConfig file. This way the directives will be saved with CONFIG_ in the final out/Arch/build/config.mk file. I would like to be able to use the existing config directives without having to rename them. And reuse the current build system without any major modifications.

[slaff]

ESP IDF uses the depends on clause to manage this ...

Ok, we can use it too. Sorry didn't know about it.

[mikee47]

Kconfig is a good solution but we now have the situation where settings are potentially replicated in at least three places: component.mk, Kconfig and README.rst.

One way to address this would be to use Kconfig as the core definition and have the build system generate data in the other two formats.