Merging Butane configs

[bgilbert]

It's convenient for users to write multiple Butane fragments and then merge them together into one config. This is awkward right now: users must separately transpile each fragment and then use Ignition's merge directive to merge them at runtime. Or, they can transpile each Butane fragment and use a wrapper script to build a top-level Butane config that inlines the Ignition fragments using merge directives.

Provide a mechanism to include Butane fragments into another Butane config and produce a unified Ignition config as output. This might be done by extending merge to allow referencing local Butane configs, transpiling each piece, and then performing the Ignition config merge at transpile time.

[dghubble]

I needed to extend terraform-provider-ct with a similar ability to add FCC v1.1.0 support and retain fragment merging in https://github.com/poseidon/terraform-provider-ct/pull/63/commits/8873e4c562197ba830ae2ec22169c35d655e1aba (called snippets there). It leaves much to be desired (i.e. its not pretty).

With no fragments, fcct's Translate is used. With fragments, the main FCC content is parsed to pick an FCC/Ignition version, then each fragment is Translated and parsed into Ignition to be able to merge Ignition Config struct's that are all of the same version. For now enforcing the FCC and any fragments are on matching versions.

I'm interested in what this might look like in fcct. Which seems to have much nicer internal primitives. My strategy of falling back through the different Ignition versions, calling Parse isn't great.

[tkarls]

I can see two main approaches. One is to have one main FCC that references others with some sort of include. Or just extend fcct to accept multiple FCC files as input. Then the tool could merge them on a yaml level before generating the ignition file (providing they have the same version etc.).

In any case. Making it simpler to divide the FCC config into several files would be appreciated!

[NickCao]

Personally prefer the first approach as it makes the dependency between fcc files trackable.

[Okeanos]

To add to this, it would be very good if the mechanism to specify local configs to be merged didn't rely on --files-dir but a potential separate --merge-dir parameter. That way physically separating a butane config and its includes from separate (common) config snippets that are reused multiple times in other cases becomes way easier to accomplish.

[bgilbert]

@Okeanos Hmm, I don't see files and merged configs as neatly divided into two distinct namespaces. For example, I think there's a reasonable argument that each merged config might want its own files-dir, at which point we'd have N separate namespaces.

The general expectation is that if you have snippets that are commonly reused, you'd render each of them into a separate Ignition config (with its own files-dir), host it at a well-defined URL, and merge it via HTTPS at provisioning time. That allows the snippets to be independently updated without rerendering the parent config.

[Okeanos]

That thought actually occurred to me later as well; forgot to update the comment, though.

[jkonecny12]

Honestly, I don't have a problem with any of these approaches, however, it would be great to have this. In general, I would like to use the Butane configs a bit as roles and playbooks in Ansible and I'm doing that but having Makefile for this is weird. I really think that this functionality should be already there.

Personally, I would go the simplest path. When the Butane found ignition -> config -> merge or similar it would first found file with the name a recursively go through all these files. Seems like the simplest and working solution to me.

[cgwalters]

It'd seem really natural to me to just support: butane foo.bu bar.bu > merged.ign.

[bgilbert]

@cgwalters We've avoided supporting that for the same reason we try to avoid command-line options affecting the semantics of the output. The instructions for assembling the final Ignition config would now reside ephemerally in your bash history, rather than persistently in a config file.

[cgwalters]

The instructions for assembling the final Ignition config would now reside ephemerally in your bash history, rather than persistently in a config file.

The idea is more one could easily write a Makefile to do this too.

[bgilbert]

Yeah, understood. Even in that case, though, the final config would now be specified in a mix of two languages/locations.

[cgwalters]

Also, I expect a lot of simple cases are e.g. butane --files-from . *.bu > merged.ign which is nearly simple enough to not even record in a Makefile.

[bgilbert]

That wildcard is problematic for another reason: the semantics of the resulting config are dependent on the merge order. The usual solution is to add sequence numbers to filenames, but users would need to know to do that.

Even if we were to support ad-hoc merging from the command line, we'd also need to support principled merging of Butane configs via Butane syntax. And it'd be a good idea to add that support first, to avoid encouraging poor config hygiene.

[tkarls]

@bgilbert Good point about the order! But how does the merge order affect the config? Does later entries overwrite earlier ones or is the first one kept and later "duplicates" discarded?

I tried to find documentation about the exact functionality of the merge key but cannot seem to find it.

[cgwalters]

The general expectation is that if you have snippets that are commonly reused, you'd render each of them into a separate Ignition config (with its own files-dir), host it at a well-defined URL, and merge it via HTTPS at provisioning time.

In my case, I have multiple machines in wildly different infrastructure (e.g. one in a public cloud, one on my home network) and I want to factor out common Ignition bits like my SSH key. They may not be able to reach a common URL, and even if they could doing this introduces a whole new level of complexity (e.g. to correctly do this you want to use verification= but if you do that, then you do need to touch each config including it when it changes).

Even in that case, though, the final config would now be specified in a mix of two languages/locations.

I'm a bit confused; are you arguing against the concept of Makefile in general? I mean my C programs are a mix of C and build rules in not-C Makefile too and I think that's been working OK :smile:

That wildcard is problematic for another reason: the semantics of the resulting config are dependent on the merge order. The usual solution is to add sequence numbers to filenames, but users would need to know to do that.

Do you have an example case in mind where someone might be depending on the merge order in a problematic way?

[jkonecny12]

The instructions for assembling the final Ignition config would now reside ephemerally in your bash history, rather than persistently in a config file.

The idea is more one could easily write a Makefile to do this too.

I'm doing that even now but it's not something I would like to do really.

[bgilbert]

@tkarls Butane doesn't currently have any docs about config merging because Butane doesn't do the merging itself. See the Ignition operator notes for more info.

@cgwalters:

In my case, I have multiple machines in wildly different infrastructure (e.g. one in a public cloud, one on my home network) and I want to factor out common Ignition bits like my SSH key. They may not be able to reach a common URL, and even if they could doing this introduces a whole new level of complexity (e.g. to correctly do this you want to use verification= but if you do that, then you do need to touch each config including it when it changes).

Yup, to be clear, client-side merging makes a lot of sense for smaller environments. Merging independent config sources at runtime is the more general case, and might make more sense in an enterprise setting where multiple teams independently maintain configs. I was arguing specifically against supporting multiple --files-dir namespaces in a single Butane run, because I don't think it makes sense to scale client-side merging to that degree.

(--files-dir is really just a security feature to prevent configs from doing arbitrary client-side directory traversal. If multiple configs are maintained by a single person or team, it's reasonable to keep all the configs in a single Git repo, and always set --files-dir to the root of the repo.)

As an aside, in your use case you may not need verification, if TLS certificate validation is good enough for your use case. You're already trusting the cloud not to tamper with the Ignition config in userdata.

I mean my C programs are a mix of C and build rules in not-C Makefile too and I think that's been working OK :slightly_smiling_face:

Sure, but merge semantics are more subtle than object linking. An analogy might be a C program with a lot of #ifdefs. To find out which parts of the code are actually compiled and run, you might need to check the Makefile to see what -D options are being passed to the compiler. But with Butane, the #ifdefs are invisible, and the behavior of the compiled code would depend on the order that the source files are specified in the Makefile. Yes, people can learn to deal with that, but it's a footgun.

Do you have an example case in mind where someone might be depending on the merge order in a problematic way?

A hardware-specific or workload-specific config might want to override pretty much anything in a site-wide config: the contents of a config file, whether to enable a systemd unit, the size of a root or data partition. (Also, if files.append is used to append directives to a config file, the order of those directives might be semantically significant.) Ignition's merge semantics are designed to encourage child configs to override fields in parent configs (or in elder siblings), exactly so that specialized configs can inherit from base configs in this way.

[jkonecny12]

A hardware-specific or workload-specific config might want to override pretty much anything in a site-wide config: the contents of a config file, whether to enable a systemd unit, the size of a root or data partition. (Also, if files.append is used to append directives to a config file, the order of those directives might be semantically significant.) Ignition's merge semantics are designed to encourage child configs to override fields in parent configs (or in elder siblings), exactly so that specialized configs can inherit from base configs in this way.

Exactly because of this I think that the Butane should benefit from the existing Ignition merge feature. What Butane should do is to look on the config and do the translation of the pointed '.bu' files in the ignition.merge directive. Basically do the translation we (at least me) are doing now in Makefile.

[alvarlagerlof]

Is there anything up-to-date on how to do this? I see mentions of makefiles but the current docs are in such a state around this topic that I cannot figure out how to do merging at all.

[bgilbert]

@alvarlagerlof Right now, the inputs to config merging are Ignition configs, not Butane configs. You can use Butane to generate both configs, but the child config referenced by the parent ignition.config.merge must be in Ignition format. The reference can be by URL or inline. For more info on config merging semantics, see here.

[alvarlagerlof]

@alvarlagerlof Right now, the inputs to config merging are Ignition configs, not Butane configs. You can use Butane to generate both configs, but the child config referenced by the parent ignition.config.merge must be in Ignition format. The reference can be by URL or inline. For more info on config merging semantics, see here.

Ah,

Thank you for the pointer.

[jkonecny12]

After that just use the standard Makefile which will check that the file does exists and solve the issues about what changed for you.

[lukasbestle]

Why this is useful

I fully agree with the proposal to combine multiple Butane files into one Ignition file at build time, for two reasons:

• It makes it possible to split long Butane files into more sensible snippets.
• It allows to reuse Butane snippets across multiple configurations for different systems.

Both of these use cases might be quite common in the real world, so I assume that a feature that allows to import/include other Butane files will be of high value to the community, especially to those who build Butane configs that are not enterprise-level but still quite complex.

Discussion summary

To get this going, I'd first like to summarize the current state of the discussion:

• There are two different possible user expectations for --files-dir:
  • Either each Butane file comes with its own --files-dir namespace. This is useful in larger deployments where different Butane/Ignition configs are maintained separately and linked to each other. But in these cases this whole feature of merging Butane configs locally at build time is not really relevant as the configs are built separately anyway.
  • Or there is a global --files-dir namespace for the whole build that will be used for all involved Butane files. This is for example how Ansible does it and useful for setups where the whole Ignition config gets built without dependencies on external config. This is also the use case where this feature of merging multiple files is missing in Butane.
• Merge order is important as multiple rules may override each other.
• An explicit syntax for child configs in Butane YAML is preferred over CLI arguments as Butane syntax makes the setup more clear and explicit.
• There are two approaches for this feature: import/include and merge. merge is already supported by Ignition and could be used with the same syntax at Butane build time. import/include on the other hand allows to insert YAML snippets at arbitrary levels. Both of these approaches are useful in the real world and one cannot be replaced with the other.

Concept

An idea how to implement both approaches (independently from each other as I wrote above):

import/include and merge via YAML tags

This could be done with a YAML tag like this:

variant: fcos
version: 1.1.0
passwd:
  users: !include 'config/users.yml'

Butane will load that other YAML file and insert the YAML structure as a child node right where it was included. The path will be resolved relative to --files-dir by the same code that is also used for files, trees etc.

A limitation of this syntax is that it's not possible to extend the imported config using native YAML syntax. The following example causes a YAML syntax error:

variant: fcos
version: 1.1.0
passwd:
  users:
    !include 'config/users.yml'
    - name: core
      ...

It could be done in theory by modifying the YAML parser, but then the Butane files wouldn't be valid YAML anymore. An alternative for valid YAML syntax would be a separate !merge tag that merges itself with its parent node (= the referenced file is included like with !include but inserted one level upwards):

variant: fcos
version: 1.1.0
passwd:
  users:
    - !merge 'config/users.yml'
    - name: core
      ...

Native Ignition merge

Because of the two different expectations for --files-dir and the two underlying use cases (see above), Butane needs to have two operating modes:

• Default mode (current behavior): Butane leaves merge declarations untouched and lets Ignition resolve them at runtime.
• Recursive mode (new --recursive flag): Butane resolves all merge declarations recursively according to the operator notes and returns a merged Ignition JSON. Butane will have the following behavior in this mode:
  • The --files-dir is used for all Butane files, including child configs. It is not possible to define a separate --files-dir per Butane file.
  • If a child config refers to another local child config (if the child config again uses merge) or to a local file or tree, this path is resolved from global the --files-dir just like in the parent config.
  • Child configs that were loaded via source (URL) are treated as external and may not refer to local child configs, files or trees. They may only use source or inline.

[dghubble]

My original aim was just for github.com/coreos/butane (the Go package) to add the Merge function for a slice of Butane snippets. It can be done (albeit painfully) by external packages (example), but would be much better within the butane package.

This would allow Butane tools to implement merging the same way. Phrased another way, we need to agree on function that can merge a list of butane snippets. Discussions about specific flag-based tools or how to expose the feature follow from that.

[lukasbestle]

My original aim was just for github.com/coreos/butane (the Go package) to add the Merge function for a slice of Butane snippets.

To be honest I can't follow. Isn't this the repo for the butane Go package (and also user-facing tool)?

Phrased another way, we need to agree on function that can merge a list of butane snippets.

Do you mean purely from the technical perspective? I.e. Butane gets two or more YAML structures to merge (however it may have gotten them) and needs to output a single YAML structure? I'd say the algorithm for this should be exactly the same one as in Ignition, provided we do use the same syntax in the end.

This leaves us with importing/including, which is a related feature but doesn't need the algorithm for config merging.

[dghubble]

Given multiple Ignition []byte contents (regardless of where they originally came from), github.com/coreos/ignition/config does not yet provide a Merge function to output a single Ignition document (absent). Individual package versions do provide a Merge function (e.g. github.com/coreos/ignition/config/v3_3), but there is not a top-level Merge to handle version introspection, etc. And handling versions correctly is important for merging. That was explored in https://github.com/coreos/butane/pull/120 as an improvement over folks having to do it themselves.

Once that is in place, Butane (the package, its users, and the cli tool) could implement a similar Merge functionality, that handles the usual Butane YAML encode/decoding. Potentially introducing new syntax into Butane to help specify the content sources seems further out when the core function is missing.

[lukasbestle]

For those who want to use including and merging right away, I have created a Makefile that uses yq to implement the first part of my suggestion:

###
# Copyright: 2021 Lukas Bestle
# License:   https://opensource.org/licenses/MIT
###

files := $(shell find files -type file)
yaml  := $(shell find . -name '*.yml')

# Final build step
dist/ignition.json: $(yaml) $(files) dist/butane.bu
	butane -d files dist/butane.bu -o dist/ignition.json

# Combines all YAML files into the merged Butane YAML
# Each merging pass resolves the `!include` and `!merge` tags:
# `!include` replaces the tag with the referenced file contents
# `!merge` merges the parent with the referenced file contents
# Multiple passes are used to resolve recursive includes/merges
dist/butane.bu: $(yaml) dist
	cp main.yml dist/.butane.bu

	for number in 1 2 3; do \
		echo "Merging pass $$number"; \
		yq eval '(.. | select(tag == "!include")) |= load(.)' -i dist/.butane.bu; \
		yq eval 'with(.. | select(tag == "!merge"); parent = (parent *+ load(.)) | del(.))' -i dist/.butane.bu; \
	done

	mv dist/.butane.bu dist/butane.bu

# Creates the dist folder if it doesn't exist
dist:
	mkdir -p dist

# Deletes all dist files
.PHONY: clean
clean:
	rm -r dist

# Spins up a temporary HTTP server to serve the ignition config
.PHONY: serve
serve: dist/ignition.json
	cd dist; python3 -m http.server

Usage

The Makefile assumes the following directory structure:

dist/
  butane.bu
  ignition.json
files/
  your-files-and-trees
main.yml
Makefile
your-custom-structure/
  groups.yml
  users.yml
  ...

Here's an example for the YAML syntax you would use:

variant: fcos
version: 1.4.0
passwd:
  groups:
    !include your-custom-structure/groups.yml
  users:
    - !merge your-custom-structure/users.yml
    - name: core
      groups:
        - wheel

Output:

variant: fcos
version: 1.4.0
passwd:
  groups:
    - name: test
  users:
    - name: core
      groups:
        - wheel
    - name: user1
      ...

[carlocorradini]

For those who want to use including and merging right away, I have created a Makefile that uses yq to implement the first part of my suggestion:

###
# Copyright: 2021 Lukas Bestle
# License:   https://opensource.org/licenses/MIT
###

files := $(shell find files -type file)
yaml  := $(shell find . -name '*.yml')

# Final build step
dist/ignition.json: $(yaml) $(files) dist/butane.bu
	butane -d files dist/butane.bu -o dist/ignition.json

# Combines all YAML files into the merged Butane YAML
# Each merging pass resolves the `!include` and `!merge` tags:
# `!include` replaces the tag with the referenced file contents
# `!merge` merges the parent with the referenced file contents
# Multiple passes are used to resolve recursive includes/merges
dist/butane.bu: $(yaml) dist
	cp main.yml dist/.butane.bu

	for number in 1 2 3; do \
		echo "Merging pass $$number"; \
		yq eval '(.. | select(tag == "!include")) |= load(.)' -i dist/.butane.bu; \
		yq eval 'with(.. | select(tag == "!merge"); parent = (parent *+ load(.)) | del(.))' -i dist/.butane.bu; \
	done

	mv dist/.butane.bu dist/butane.bu

# Creates the dist folder if it doesn't exist
dist:
	mkdir -p dist

# Deletes all dist files
.PHONY: clean
clean:
	rm -r dist

# Spins up a temporary HTTP server to serve the ignition config
.PHONY: serve
serve: dist/ignition.json
	cd dist; python3 -m http.server

Usage

The Makefile assumes the following directory structure:

dist/
  butane.bu
  ignition.json
files/
  your-files-and-trees
main.yml
Makefile
your-custom-structure/
  groups.yml
  users.yml
  ...

Here's an example for the YAML syntax you would use:

variant: fcos
version: 1.4.0
passwd:
  groups:
    !include your-custom-structure/groups.yml
  users:
    - !merge your-custom-structure/users.yml
    - name: core
      groups:
        - wheel

Output:

variant: fcos
version: 1.4.0
passwd:
  groups:
    - name: test
  users:
    - name: core
      groups:
        - wheel
    - name: user1
      ...

We really need the merging feature... 😅

Thanks @lukasbestle for sharing your work!

[LorbusChris]

Here's another example of how to (naively) merge Butane config snippets: https://github.com/LorbusChris/butane-config-template

[carlocorradini]

Here's another example of how to (naively) merge Butane config snippets: https://github.com/LorbusChris/butane-config-template

So useful! Thanks!