Support for multiple configuration sources

[paynejacob]

Currently C3OS will search a few pre-defined directories for configuration and select the first configuration file it finds. It would be nice if c3os layers all the configuration files it finds into a single configuration. The rules for how configuration is layers would need to be clearly defined but this could have a few benefits as I will outline below.

Thoughts on merge strategy

The challenge of supporting multiple configurations is how to determine which configuration fields take precedence over others. A simple approach would be to give precedence to the file names with a higher lexical order within a directory and higher precedence to directories based on the order they are defined. Non empty values will always take precedence over empty or missing keys.

This strategy is easy to understand and implement but would be less ideal for fields like the elemental stages. In this case it would make more sense to append array values rather that replacing existing arrays. Given how tightly coupled c3os is to elemental style cloud init a special exception could be made for the stages key and this would likely get us to a usable implementation fairly quickly.

A more complex implementation could allow users to define a mask for their cloud configuration. This would require significant work around validation and making it clear to the user what rules were applied and how the final cloud configuration was rendered but would give the flexibility for users to determine what configurations take what precedence. Plugins could also include a default mask that c3os would apply.

Example:

# cloud-config
install:
  auto: true
  device: /dev/sda

cluster:
  cluster_token: ""

stages:
  initramfs:
    -  environment:
           foo: bar
 ---
 # mask
 install: overwrite
 
 cluster: overwriteIfEmpty
 
 stages:
   *: append
 

Use Cases

Organization

As c3os becomes more pluggable the number of configuration sections the cloud init will continue to grow. It can be helpful for users to break the configuration into multiple files to make it easier to find configuration for their particular component.

Overrides

Given the extensible nature of c3os and use of layering in it's architecture users may want to define a default set of options for their generic base image and override options for particular features. Users may create a common install image that performs a simplistic installation. They could then mount cloud config via IPXE for specific MAC addresses to provide alternative settings for special case deployments.

Extensibility

Given the plugin nature of c3os it stands to reason that different layers will want to include default configuration. Users will also want to use the best configuration method for their use case. Including ca certs as a cloud config file in an image makes sense but configuring the device would be better served by IPXE as it can have some awareness of the hardware. I may also want to pull a cluster plugin configuration with a custom install plugin. This would be easiest to accomplish by each system writing a different cloud config to a know directory and c3os flattening them.