Remove conditional compilation from `early-boot-config`

[zmrow]

Issue number:

Closes #3616

Description of changes: This PR is best read in commit order! Commits are fairly small and self-contained, but there are a few of them. :)

The strategy for this PR is largely laid out in #3616, but the changes in this PR remove all conditional compilation from early-boot-config by using discrete "user data provider" binaries for each applicable user data source.

Logic for fetching user data was largely "lifted and shifted" to discrete modules for each source. A new trait UserDataProvider was added, with each user data source implementing said trait (local file, IMDS, guestinfo, etc).

Each user data source is bundled into it's own binary within the early-boot-config package. Each binary is packaged separately with its own package in the packages/ directory. Each package includes two subpackages, a -client package which includes just the binary, and a -provider package that symlinks the binary into a .d-style directory in the proper order. Ordering of sources stays identical to the ordering we have today.

All of the user data provider packages are bundled into an early-boot-config-providers metapackage that contains subpackages for each variant type: AWS, VMware, metal. The variant type determines the proper providers. Each variant then consumes the corresponding metapackage, i.e aws-k8s-1.28 will consume the aws-data-providers package.

At runtime, early-boot-config consults said .d directory (/usr/share/early-boot-config/data-providers.d) and runs the binaries in the proper order. If user data is found at the source, the binary returns a serialized SettingsJson object on stdout, along with a successful exit code. If no user data is found, but no problems were encountered, it exits successfully, but outputs nothing. If any errors are encountered, the binary exits unsuccessful and prints its error.

Similar to today's logic, early-boot-config submits the user data to the API in order.

Example output, showing the proper order and provider log output on an aws-dev variant:

bash-5.1# journalctl -u early-boot-config
Dec 04 18:41:07 localhost systemd[1]: Starting Bottlerocket userdata configuration system...
Dec 04 18:41:07 localhost early-boot-config[1017]: [2023-12-04T18:41:07Z INFO  early_boot_config] early-boot-config started
Dec 04 18:41:07 localhost early-boot-config[1017]: [2023-12-04T18:41:07Z INFO  early_boot_config] Gathering user data providers
Dec 04 18:41:07 localhost early-boot-config[1017]: [2023-12-04T18:41:07Z INFO  early_boot_config] Provider '10-local-defaults': [2023-12-04T18:41:07Z INFO  early_boot_config::provider] /local/user-data-defaults.toml does not exist, not using it
Dec 04 18:41:07 localhost early-boot-config[1017]: [2023-12-04T18:41:07Z INFO  early_boot_config] No user data found via /usr/share/early-boot-config/data-providers.d/10-local-defaults
Dec 04 18:41:07 localhost early-boot-config[1017]: [2023-12-04T18:41:07Z INFO  early_boot_config] Provider '20-local': [2023-12-04T18:41:07Z INFO  early_boot_config::provider] /var/lib/bottlerocket/user-data.toml does not exist, not using it
Dec 04 18:41:07 localhost early-boot-config[1017]: [2023-12-04T18:41:07Z INFO  early_boot_config] No user data found via /usr/share/early-boot-config/data-providers.d/20-local
Dec 04 18:41:07 localhost early-boot-config[1017]: [2023-12-04T18:41:07Z INFO  early_boot_config] Provider '30-ec2-identity-doc': [2023-12-04T18:41:07Z INFO  early_boot_config::provider::ec2_identity_doc] Using IMDS for region
Dec 04 18:41:07 localhost early-boot-config[1017]: [2023-12-04T18:41:07Z INFO  early_boot_config] Provider '30-ec2-identity-doc': [2023-12-04T18:41:07Z INFO  imdsclient] Received dynamic/instance-identity/document
Dec 04 18:41:07 localhost early-boot-config[1017]: [2023-12-04T18:41:07Z INFO  early_boot_config] Found user data via EC2 instance identity document, sending to API
Dec 04 18:41:07 localhost early-boot-config[1017]: [2023-12-04T18:41:07Z INFO  early_boot_config] Provider '40-ec2-imds': [2023-12-04T18:41:07Z INFO  early_boot_config::provider::ec2_imds] Fetching user data from IMDS
Dec 04 18:41:07 localhost early-boot-config[1017]: [2023-12-04T18:41:07Z INFO  early_boot_config] Provider '40-ec2-imds': [2023-12-04T18:41:07Z INFO  imdsclient] Received user-data
Dec 04 18:41:07 localhost early-boot-config[1017]: [2023-12-04T18:41:07Z INFO  early_boot_config] Found user data via EC2 IMDS, sending to API
Dec 04 18:41:07 localhost early-boot-config[1017]: [2023-12-04T18:41:07Z INFO  early_boot_config] Provider '50-local-overrides': [2023-12-04T18:41:07Z INFO  early_boot_config::provider] /local/user-data-overrides.toml does not exist, not using it
Dec 04 18:41:07 localhost early-boot-config[1017]: [2023-12-04T18:41:07Z INFO  early_boot_config] No user data found via /usr/share/early-boot-config/data-providers.d/50-local-overrides
Dec 04 18:41:07 localhost systemd[1]: Finished Bottlerocket userdata configuration system.

An example from VMware showing proper sources (CD-ROM and guestinfo both include user data):

bash-5.1# journalctl -u early-boot-config
Dec 07 20:38:51 localhost systemd[1]: Starting Bottlerocket userdata configuration system...
Dec 07 20:38:51 localhost early-boot-config[1697]: [2023-12-07T20:38:51Z INFO  early_boot_config] early-boot-config started
Dec 07 20:38:51 localhost early-boot-config[1697]: [2023-12-07T20:38:51Z INFO  early_boot_config] Gathering user data providers
Dec 07 20:38:51 localhost early-boot-config[1697]: [2023-12-07T20:38:51Z INFO  early_boot_config] Provider '10-local-defaults': [2023-12-07T20:38:51Z INFO  early_boot_config::provider] /local/user-data-defaults.toml does not exist, not using it
Dec 07 20:38:51 localhost early-boot-config[1697]: [2023-12-07T20:38:51Z INFO  early_boot_config] No user data found via /usr/share/early-boot-config/data-providers.d/10-local-defaults
Dec 07 20:38:51 localhost early-boot-config[1697]: [2023-12-07T20:38:51Z INFO  early_boot_config] Provider '20-local': [2023-12-07T20:38:51Z INFO  early_boot_config::provider] /var/lib/bottlerocket/user-data.toml does not exist, not using it
Dec 07 20:38:51 localhost early-boot-config[1697]: [2023-12-07T20:38:51Z INFO  early_boot_config] No user data found via /usr/share/early-boot-config/data-providers.d/20-local
Dec 07 20:38:51 localhost early-boot-config[1697]: [2023-12-07T20:38:51Z INFO  early_boot_config] Provider '30-vmware-cd-rom': [2023-12-07T20:38:51Z INFO  early_boot_config::provider::vmware_cd_rom] Attempting to retrieve user data from mounted CD-ROM
Dec 07 20:38:51 localhost early-boot-config[1697]: [2023-12-07T20:38:51Z INFO  early_boot_config] Provider '30-vmware-cd-rom': [2023-12-07T20:38:51Z INFO  early_boot_config::provider::vmware_cd_rom] '/media/cdrom/ovf_env.xml' exists, using it
Dec 07 20:38:51 localhost early-boot-config[1697]: [2023-12-07T20:38:51Z INFO  early_boot_config] Found user data via CD-ROM, sending to API
Dec 07 20:38:51 localhost early-boot-config[1697]: [2023-12-07T20:38:51Z INFO  early_boot_config] Provider '40-vmware-guestinfo': [2023-12-07T20:38:51Z INFO  early_boot_config::provider::vmware_guestinfo] Attempting to retrieve user data via guestinfo interface
Dec 07 20:38:51 localhost early-boot-config[1697]: [2023-12-07T20:38:51Z INFO  early_boot_config] Provider '40-vmware-guestinfo': [2023-12-07T20:38:51Z INFO  early_boot_config::provider::vmware_guestinfo] Found user data encoding: base64
Dec 07 20:38:51 localhost early-boot-config[1697]: [2023-12-07T20:38:51Z INFO  early_boot_config] Provider '40-vmware-guestinfo': [2023-12-07T20:38:51Z INFO  early_boot_config::provider::vmware_guestinfo] Decoding user data
Dec 07 20:38:51 localhost early-boot-config[1697]: [2023-12-07T20:38:51Z INFO  early_boot_config] Found user data via guestinfo, sending to API
Dec 07 20:38:51 localhost early-boot-config[1697]: [2023-12-07T20:38:51Z INFO  early_boot_config] Provider '50-local-overrides': [2023-12-07T20:38:51Z INFO  early_boot_config::provider] /local/user-data-overrides.toml does not exist, not using it
Dec 07 20:38:51 localhost early-boot-config[1697]: [2023-12-07T20:38:51Z INFO  early_boot_config] No user data found via /usr/share/early-boot-config/data-providers.d/50-local-overrides
Dec 07 20:38:51 localhost systemd[1]: Finished Bottlerocket userdata configuration system.

Testing done:

• [x] Ensure ordering is correct for each variant type (AWS, VMware, metal)
  • [x] AWS (aws-dev, aws-k8s-1.27, aws-k8s-1.28)
  • [x] VMware (vmware-k8s-1.28)
  • [x] metal (metal-dev)
• [x] Test with multiple sources present with distinct settings (aws-k8s-1.28): Observe settings from EC2 instance identity doc, and user data from IMDS both populated in the API.
• [x] Test with multiple sources present with setting overrides (metal-dev): Set settings.motd via user-data, but overrode it with /local/user-data-overrides.toml. Observe the overriden value present in the API settings.
• [x] Test all providers
  • [x] Files
  • [x] IMDS
  • [x] Identity doc
  • [x] CD-ROM (used an ovf_env.xml file inside an ISO attached to VMware VM)
  • [x] Guestinfo

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[zmrow]

^ Added additional build-dependencies to the variant Cargo.toml's.

[zmrow]

^ Remove offending --bin from early-boot-config's %build line in os.spec that was causing builds to fail

[zmrow]

^ In the os package, build early-boot-config separately with the --bin flag. This ensures only that binary is built, and not the user data providers which may have architecture-dependent components.

[zmrow]

^ Ensure VMware guestinfo provider is only included for x86_64

[zmrow]

^ Added an early-boot-config package, moving it out of the os package. Renamed all the related user data provider packages with the prefix early-boot-config

[bcressey]

We need to wind all of this into a single early-boot-config package with sub-packages.

Some of the tricks that @jmt-lab is using in #3700 should help, like Supplements and using a Lua scriptlet to create a symlink.

[webern]

@sam-berning you could get this rebased as a starting point!

[webern]

We need to wind all of this into a single early-boot-config package with sub-packages.

Some of the tricks that @jmt-lab is using in #3700 should help, like Supplements and using a Lua scriptlet to create a symlink.

@bcressey we want to make sure that we compile all of the code (barring VMWare bits when building arm) when we cd sources && cargo build and also when we cargo make for any variant. Just making sure your packaging suggestion doesn't reintroduce conditional compilation in any way. In #3700 we ended up building twice in the RPM.spec because we didn't actually remove conditional compilation! :sweat_smile: , but with early-boot-config, as written in this PR currently, we have. We want to make sure not to lose sight of that prize when contemplating packaging strategy.

[bcressey]

@webern - absolutely, I just don't want the packaging scattered across nine spec files.

In some sense each spec is "free" but only if future changes are largely decoupled, and if they don't all need to be understood together to form a mental model of the behavior on the installed system.

One of my packaging regrets is splitting Docker into 4 packages (now down to 3) when it really should have been 1 or 2 all along.

The current scattering of systemd-related functionality across release, os, and host-ctr is another concern I have. That's a symptom of the larger problem is that "os" is too large and monolithic, and ends up containing things because of Rust compile times rather than sensible modular boundaries.

We're on a path to fixing that, and this work is part of the fix, but I don't want to go to a different extreme of micro-packages in the process.

[webern]

We're on a path to fixing that, and this work is part of the fix, but I don't want to go to a different extreme of micro-packages in the process.

One thing to think about is kit design and whether or not you would want sub-packages to belong to different kits.

[sam-berning]

^ rebased on latest develop. Zac's original branch state is backed up to https://github.com/sam-berning/bottlerocket/tree/zmrow-ebc-src-bin

[sam-berning]

^ updated ebc to run providers in parallel but apply their results in order, and packaged everything all in one rpm

[sam-berning]

Looks like the vmw_backdoor crate only builds for x86, so the vmware_guestinfo provider at least will need to be split out into a different rpm. I think I'll extract the early-boot-config-vmware package out to its own rpm.

[sam-berning]

^ actually ended up creating a dummy version of the vmware-guestinfo provider for arm so that we can keep everything in one package for now.

this is technically adding more conditional compilation, but I think it's still in the right direction. I'll mess around with pulling the user data providers into separate crates from each other and from early-boot-config so their Cargo.tomls aren't coupled and hopefully then we can remove conditional compilation altogether

[sam-berning]

^ rebased on latest develop and separated the user data provider interface from early-boot-config and from the data providers themselves. also conditionally packaged the vmware data providers only if we're building for x86.

[sam-berning]

the commits probably need to be squashed a bit before we merge but I wanted to keep as much of the history on this PR as possible while we're reviewing. should be still readable commit-by-commit.

[sam-berning]

^ addressed comments

[sam-berning]

^ fixed merge conflicts and addressed @bcressey's comments.

Tested all sets of user data providers by building aws-dev, vmware-dev, and metal-dev. For each variant, I injected local files with a custom rpm2img script. Verified that the user data looked as expected on all variants, and checked the journalctl logs to see that all providers picked up the provided user data and applied it in the correct order.

Testing report:

metal:

bash-5.1# journalctl | grep early-boot-config
Apr 03 23:00:23 localhost early-boot-config[1219]: [2024-04-03T23:00:23Z INFO  early_boot_config] early-boot-config started
Apr 03 23:00:23 localhost early-boot-config[1219]: [2024-04-03T23:00:23Z INFO  early_boot_config] Gathering user data providers
Apr 03 23:00:23 localhost early-boot-config[1219]: [2024-04-03T23:00:23Z INFO  early_boot_config] Provider '10-local-defaults': [2024-04-03T23:00:23Z INFO  early_boot_config_provider::provider] '/local/user-data-defaults.toml' exists, using it
Apr 03 23:00:23 localhost early-boot-config[1219]: [2024-04-03T23:00:23Z INFO  early_boot_config] Found user data via user data from /local/user-data-defaults.toml, sending to API
Apr 03 23:00:23 localhost early-boot-config[1219]: [2024-04-03T23:00:23Z INFO  early_boot_config] Provider '20-local-user-data': [2024-04-03T23:00:23Z INFO  early_boot_config_provider::provider] '/var/lib/bottlerocket/user-data.toml' exists, using it
Apr 03 23:00:23 localhost early-boot-config[1219]: [2024-04-03T23:00:23Z INFO  early_boot_config] Found user data via user data from /var/lib/bottlerocket/user-data.toml, sending to API
Apr 03 23:00:23 localhost early-boot-config[1219]: [2024-04-03T23:00:23Z INFO  early_boot_config] Provider '99-local-overrides': [2024-04-03T23:00:23Z INFO  early_boot_config_provider::provider] '/local/user-data-overrides.toml' exists, using it
Apr 03 23:00:23 localhost early-boot-config[1219]: [2024-04-03T23:00:23Z INFO  early_boot_config] Found user data via user data from /local/user-data-overrides.toml, sending to API

VMware:

bash-5.1# journalctl | grep early-boot-config
Apr 04 00:18:16 localhost early-boot-config[1690]: [2024-04-04T00:18:16Z INFO  early_boot_config] early-boot-config started
Apr 04 00:18:16 localhost early-boot-config[1690]: [2024-04-04T00:18:16Z INFO  early_boot_config] Gathering user data providers
Apr 04 00:18:17 localhost early-boot-config[1690]: [2024-04-04T00:18:17Z INFO  early_boot_config] Provider '10-local-defaults': [2024-04-04T00:18:17Z INFO  early_boot_config_provider::provider] '/local/user-data-defaults.toml' exists, using it
Apr 04 00:18:17 localhost early-boot-config[1690]: [2024-04-04T00:18:17Z INFO  early_boot_config] Found user data via user data from /local/user-data-defaults.toml, sending to API
Apr 04 00:18:17 localhost early-boot-config[1690]: [2024-04-04T00:18:17Z INFO  early_boot_config] Provider '20-local-user-data': [2024-04-04T00:18:17Z INFO  early_boot_config_provider::provider] '/var/lib/bottlerocket/user-data.toml' exists, using it
Apr 04 00:18:17 localhost early-boot-config[1690]: [2024-04-04T00:18:17Z INFO  early_boot_config] Found user data via user data from /var/lib/bottlerocket/user-data.toml, sending to API
Apr 04 00:18:17 localhost early-boot-config[1690]: [2024-04-04T00:18:17Z INFO  early_boot_config] Provider '30-vmware-cd-rom': [2024-04-04T00:18:17Z INFO  vmware_cd_rom_user_data_provider] Attempting to retrieve user data from mounted CD-ROM
Apr 04 00:18:17 localhost early-boot-config[1690]: [2024-04-04T00:18:17Z INFO  early_boot_config] Provider '30-vmware-cd-rom': [2024-04-04T00:18:17Z INFO  vmware_cd_rom_user_data_provider] '/media/cdrom/user-data' exists, using it
Apr 04 00:18:17 localhost early-boot-config[1690]: [2024-04-04T00:18:17Z INFO  early_boot_config] Found user data via CD-ROM, sending to API
Apr 04 00:18:17 localhost early-boot-config[1690]: [2024-04-04T00:18:17Z INFO  early_boot_config] Provider '40-vmware-guestinfo': [2024-04-04T00:18:17Z INFO  vmware_guestinfo_user_data_provider::x86_64] Attempting to retrieve user data via guestinfo interface
Apr 04 00:18:17 localhost early-boot-config[1690]: [2024-04-04T00:18:17Z INFO  early_boot_config] Provider '40-vmware-guestinfo': [2024-04-04T00:18:17Z INFO  vmware_guestinfo_user_data_provider::x86_64] Found user data encoding: base64
Apr 04 00:18:17 localhost early-boot-config[1690]: [2024-04-04T00:18:17Z INFO  early_boot_config] Provider '40-vmware-guestinfo': [2024-04-04T00:18:17Z INFO  vmware_guestinfo_user_data_provider::x86_64] Decoding user data
Apr 04 00:18:17 localhost early-boot-config[1690]: [2024-04-04T00:18:17Z INFO  early_boot_config] Found user data via guestinfo, sending to API
Apr 04 00:18:17 localhost early-boot-config[1690]: [2024-04-04T00:18:17Z INFO  early_boot_config] Provider '99-local-overrides': [2024-04-04T00:18:17Z INFO  early_boot_config_provider::provider] '/local/user-data-overrides.toml' exists, using it
Apr 04 00:18:17 localhost early-boot-config[1690]: [2024-04-04T00:18:17Z INFO  early_boot_config] Found user data via user data from /local/user-data-overrides.toml, sending to API

AWS:

bash-5.1#  journalctl | grep early-boot-config
Apr 03 22:30:51 localhost early-boot-config[1234]: [2024-04-03T22:30:51Z INFO  early_boot_config] early-boot-config started
Apr 03 22:30:51 localhost early-boot-config[1234]: [2024-04-03T22:30:51Z INFO  early_boot_config] Gathering user data providers
Apr 03 22:30:51 localhost early-boot-config[1234]: [2024-04-03T22:30:51Z INFO  early_boot_config] Provider '10-local-defaults': [2024-04-03T22:30:51Z INFO  early_boot_config_provider::provider] '/local/user-data-defaults.toml' exists, using it
Apr 03 22:30:51 localhost early-boot-config[1234]: [2024-04-03T22:30:51Z INFO  early_boot_config] Found user data via user data from /local/user-data-defaults.toml, sending to API
Apr 03 22:30:51 localhost early-boot-config[1234]: [2024-04-03T22:30:51Z INFO  early_boot_config] Provider '20-local-user-data': [2024-04-03T22:30:51Z INFO  early_boot_config_provider::provider] '/var/lib/bottlerocket/user-data.toml' exists, using it
Apr 03 22:30:51 localhost early-boot-config[1234]: [2024-04-03T22:30:51Z INFO  early_boot_config] Found user data via user data from /var/lib/bottlerocket/user-data.toml, sending to API
Apr 03 22:30:51 localhost early-boot-config[1234]: [2024-04-03T22:30:51Z INFO  early_boot_config] Provider '30-ec2-identity-doc': [2024-04-03T22:30:51Z INFO  ec2_identity_doc_user_data_provider] Using IMDS for region
Apr 03 22:30:51 localhost early-boot-config[1234]: [2024-04-03T22:30:51Z INFO  early_boot_config] Provider '30-ec2-identity-doc': [2024-04-03T22:30:51Z INFO  imdsclient] Received dynamic/instance-identity/document
Apr 03 22:30:51 localhost early-boot-config[1234]: [2024-04-03T22:30:51Z INFO  early_boot_config] Found user data via EC2 instance identity document, sending to API
Apr 03 22:30:51 localhost early-boot-config[1234]: [2024-04-03T22:30:51Z INFO  early_boot_config] Provider '40-ec2-imds': [2024-04-03T22:30:51Z INFO  ec2_imds_user_data_provider] Fetching user data from IMDS
Apr 03 22:30:51 localhost early-boot-config[1234]: [2024-04-03T22:30:51Z INFO  early_boot_config] Provider '40-ec2-imds': [2024-04-03T22:30:51Z INFO  imdsclient] Received user-data
Apr 03 22:30:51 localhost early-boot-config[1234]: [2024-04-03T22:30:51Z INFO  early_boot_config] Found user data via EC2 IMDS, sending to API
Apr 03 22:30:51 localhost early-boot-config[1234]: [2024-04-03T22:30:51Z INFO  early_boot_config] Provider '99-local-overrides': [2024-04-03T22:30:51Z INFO  early_boot_config_provider::provider] '/local/user-data-overrides.toml' exists, using it
Apr 03 22:30:51 localhost early-boot-config[1234]: [2024-04-03T22:30:51Z INFO  early_boot_config] Found user data via user data from /local/user-data-overrides.toml, sending to API

[sam-berning]

^ rebased once more