Automatically create storage config from source directory

[PhrozenByte]

/kind enhancement

I'm using Butane to provision some servers with rather complex setups and unfortunately I've to say that Butane's way of handling files, directories and links makes things quite hard right now.

Adding new files, directories and links to a server is probably the most common task when provisioning a server, however, right now one has to add every single file manually to config.bu. The biggest problem are files, as one has to either inline their contents (making the config unreadable pretty fast), or one has to create the source file separately and set a source path in config.bu. Luckily one doesn't have to add full directory trees to config.bu, because Ignition will silently create missing directories anyway - unless we want the directories not to be owned by root. Then we must add the full directory tree, too.

Here's an example of such a config.bu - that needs to be multiplied a few times, once per rootless container (you don't have to actually check the config, it doesn't matter, it's just an example).

variant: fcos
version: 1.4.0
passwd:
  groups:
    - name: acme
      gid: 2000
    - name: acme_croot
      gid: 100000
  users:
    - name: acme
      uid: 2000
      primary_group: acme
      home_dir: /srv/containers/acme
    - name: acme_croot
      uid: 100000
      primary_group: acme_croot
      home_dir: /srv/containers/acme
      no_create_home: true
      shell: /usr/bin/nologin
      system: true
storage:
  directories:
    - path: /srv/containers/acme
      mode: 0700
      user: { name: "acme" }
      group: { name: "acme" }
    - path: /srv/containers/acme/data
      user: { name: "acme" }
      group: { name: "acme" }
    - path: /srv/containers/acme/config
      user: { name: "acme" }
      group: { name: "acme" }
    - path: /srv/containers/acme/.config
      user: { name: "acme" }
      group: { name: "acme" }
    - path: /srv/containers/acme/.config/systemd
      user: { name: "acme" }
      group: { name: "acme" }
    - path: /srv/containers/acme/.config/sxstemd/user
      user: { name: "acme" }
      group: { name: "acme" }
    - path: /srv/containers/acme/.config/systemd/user/multi-user.target.wants
      user: { name: "acme" }
      group: { name: "acme" }
  files:
    - path: /etc/subuid
      append:
        - inline: |
            acme:100000:65536
    - path: /etc/subgid
      append:
        - inline: |
            acme:100000:65536
    - path: /var/lib/systemd/linger/acme
    - path: /srv/containers/acme/.config/systemd/user/container-acme.service
      user: { name: "acme" }
      group: { name: "acme" }
      overwrite: true
      contents:
        inline: |
          [Unit]
          Description=Podman container-acme.service
          Wants=network-online.target
          After=network-online.target
          RequiresMountsFor=%t/containers
          
          [Service]
          Environment=PODMAN_SYSTEMD_UNIT=%n
          Restart=on-failure
          TimeoutStopSec=70
          ExecStartPre=/bin/rm -f %t/%n.ctr-id
          ExecStart=/usr/bin/podman run --cidfile=%t/%n.ctr-id --sdnotify=conmon --cgroups=no-conmon --replace -dt --name acme --pull always --uidmap 0:1:65536 --uidmap 65536:0:1 --gidmap 0:1:65536 --gidmap 65536:0:1 --mount type=bind,src=/srv/containers/acme/data,dst=/var/local/acme --mount type=bind,src=/srv/containers/acme/config,dst=/etc/acme ghcr.io/sgsgermany/acme:latest
          ExecStop=/usr/bin/podman stop --ignore --cidfile=%t/%n.ctr-id
          Type=notify
          NotifyAccess=all
          
          [Install]
          WantedBy=multi-user.target
  links:
    - path: /srv/containers/acme/.config/systemd/user/multi-user.target.wants/container-acme.service
      user: { name: "acme" }
      group: { name: "acme" }
      overwrite: true
      target: ../container-acme.service

This is very repetitive and error-prone (did you see the typo for /srv/containers/acme/.config/systemd/user?).

Thus I'd like to suggest to add a new feature allowing Butane to automatically create storage configs from a source directory. For this we need two additional command line options, --storage-src-dir and --storage-config-file (or similar). --storage-src-dir takes a path to a directory, --storage-config-file a file name (defaults to subconfig.bu, or similar).

Butane iterates all files, directories and links in --storage-src-dir and automatically creates entries for these files in the resulting Ignition config, adding path, contents (for files only) and target (for links only) respectively. File ownership and permissions are ignored. This allows users to add files to their Ignition config by simply creating the necessary files and directories in the --storage-src-dir directory. For all other parameters (like user, group, mode and overwrite), of a directory, file or link, one uses magic files like subconfig.bu, matching the --storage-config-file command line option.

Here's an example of a subconfig.bu stored at …/srv/containers/acme/subconfig.bu

variant: fcos-sub
version: 1.5.0
directories:
  - path: /
    mode: 0700
  - path: "*"
    user: { name: "acme" }
    group: { name: "acme" }
files:
  - path: "*"
    user: { name: "acme" }
    group: { name: "acme" }
    overwrite: true
links:
  - path: "*"
    user: { name: "acme" }
    group: { name: "acme" }
    overwrite: true

The config.bu is now way simpler.

variant: fcos
version: 1.5.0
passwd:
  groups:
    - name: acme
      gid: 2000
    - name: acme_croot
      gid: 100000
  users:
    - name: acme
      uid: 2000
      primary_group: acme
      home_dir: /srv/containers/acme
    - name: acme_croot
      uid: 100000
      primary_group: acme_croot
      home_dir: /srv/containers/acme
      no_create_home: true
      shell: /usr/bin/nologin
      system: true
storage:
  files:
    - path: /etc/subuid
      append:
        - inline: |
            acme:100000:65536
    - path: /etc/subgid
      append:
        - inline: |
            acme:100000:65536

subconfig.bu files are used to merge their config into the specification of files, directories and links matching the path pattern below this point. The path patterns are just usual glob patterns, like e.g. in .gitignore files. So, for example, the path: / directory config adds mode: 0700 to the specification of /srv/containers/acme. Since the directory pattern path: "*" matches everything, it ensures that all directories below /srv/containers/acme (including /srv/containers/acme) are owned by acme. The same for files and links, which additionally set overwrite: true.

I wrote a pre-processor for Butane with Python (unfortunately I don't know Go...) that implements exactly this (it also implements #118; no public release yet). It makes things way easier. What do you guys think about this?

[bgilbert]

Have you tried the storage.trees section? It's intended for exactly this use case.

[PhrozenByte]

To be honest, I wasn't aware that this exists :astonished:

However, checking it out now it definitely seems to make a few things easier, especially including file contents, but it doesn't really solve the issue. The only line omitted with storage.trees is that of file /var/lib/systemd/linger/acme - as it's the only file that doesn't need custom file permissions and/or ownership. Besides file content inclusion - that evidently has been solved already - the core of my suggestion is path pattern matching, allowing one to apply rules to multiple paths at once. And #118, but that isn't new.

Now knowing that storage.trees exists, I'd suggest not to use separate subconfig.bu files, but rather extend storage.trees with storage.trees[].file_pattern, storage.trees[].directory_pattern, and storage.trees[].link_pattern, doing the same as what my previously suggested subconfig.bu would do.

Files owned by root aren't really an issue for me. It's other user's files, because we heavily use rootless containers.

[bgilbert]

We kept the initial trees implementation minimal for simplicity, with the assumption that we could add more features as needed. I actually have an old branch with half an implementation of this:

storage:
  trees:
    - local: foo
      path: /
      user:
        id: <>
        name: <>
      group:
        id: <>
        name: <>

And we could have file_mode and directory_mode fields as well. The idea is that if you need different defaults for different subdirectories, you can express that by specifying multiple trees. How well would that work for you? I'd prefer to avoid pattern-matching if possible, since then we'd need to deal with e.g. conflicts among patterns and between patterns and overrides.

[PhrozenByte]

Yeah, this sounds very good :+1:

This way I could have a root user tree and a rootless user tree per container. It doesn't solve everything, like a few containers require some more patterns (e.g. files for podman secret that need special ownership and modes), so I might end up with three or four trees max for some containers and a handful of separate file specs, but with storage.trees[].user, group, file_mode, directory_mode and overwrite (since I usually preserve /var the paths below /srv often exist already, so I'd say we also need overwrite) it would definitely get manageable.

Conflicts with multiple patterns matching was no big deal for my Python implementation by the way, the rules simply stack (just a fyi, I'm totally file with the solution above). #118 actually was a bigger deal considering conflicts, as all rootless containers require adding their users to /etc/subuid and /etc/subgid. I solved it by merging multiple append configs for the same path, but bailing when any following spec contains a content config or a different user, group, mode, or overwrite config.

By the way: I've just released said Python script. It's called mbutane (for "merge butane") and can be found here: https://github.com/PhrozenByte/mbutane. Just for reference... I think we found a different solution better matching Butane here.

[lukaspieper]

Hi there,

I usually avoid writing comments that are basically a "+1" but as this issue hasn't seen any activity recently, I hope it's fine to point out that I would love to have more options for trees. Having overwrite in a first step would be amazing.

[SecT0uch]

This is exactly what we need for rootless containers:

We kept the initial trees implementation minimal for simplicity, with the assumption that we could add more features as needed. I actually have an old branch with half an implementation of this:

storage:
  trees:
    - local: foo
      path: /
      user:
        id: <>
        name: <>
      group:
        id: <>
        name: <>

We currently have to specify each single file and directory if we want them to be owned by an user.

@bgilbert any chance this will be implemented ?

[PhrozenByte]

We currently have to specify each single file and directory if we want them to be owned by an user.

I know it's a little clumsy, but till (if ever) this is implemented, you might want to take a look at https://github.com/PhrozenByte/mbutane. It implements just this using a Python wrapper around Butane. It surely is no perfect solution and any native solution would be way superior, but it works and makes my life easier for a few years now.

[SecT0uch]

I think we would rather need to have this natively in the main butane project TBH