bootc switch does not tell me when it ignores my /etc changes

[stefwalter]

I've made changes to /etc in my Container file:

Tag quay.io/swalter/monday:test1

RUN echo 'test1' > /etc/blah

Tag quay.io/swalter/monday:test2

RUN echo 'test2' > /etc/blah

If I modify /etc/blah on a deployed system, and then run bootc switch the system ignores the changes represented in my Containerfile, but doesn't tell me about this. This doesn't represent typical behavior I expect from a container. I expect the changes in /etc to take effect, or at a minimum for bootc to tell me when this isn't the case:

# cat /etc/blah 
test2
# bootc status | grep image
  image:
    image: quay.io/swalter/monday:test2
    image:
      image:
        image: quay.io/swalter/monday:test2
      imageDigest: sha256:f1ec8375caf4ec43d5dd37f26dcfcca8eb7279dbee1a368f55b394e551096112
# echo "test different" >> /etc/blah
# bootc switch quay.io/swalter/monday:test1
No changes in quay.io/swalter/monday:test1 => sha256:09f60a3d90f9604c5802eca575e56df262b55f78f4cd20269c0aaae81c870ce7
Queued for next boot: ostree-unverified-registry:quay.io/swalter/monday:test1
  Version: stream9.20240319.0
  Digest: sha256:09f60a3d90f9604c5802eca575e56df262b55f78f4cd20269c0aaae81c870ce7
# reboot 
...
# cat /etc/blah 
test2
test different

[cgwalters]

I did https://github.com/containers/bootc/pull/441 to try to expand on this more, but...yes. There is ostree admin config-diff but I think that one is ripe to be better fleshed out for bootc.

Bigger picture, dealing with drift in /etc is complex because there are important use cases that require machine-specific state, and that gets into the problem of distinguishing that stuff from "accidental" changes.

[cgwalters]

I added a bit more to the docs on this too, but a challenge here in a nutshell is everyone wants "just exactly the needed state and no more"...that's the hard part :smile:

I saw https://grahamc.com/blog/erase-your-darlings/ go by a while ago and thought it was a great example of this "OK well, we can't keep track of everything writing to /etc so we'll make a new /persist directory". (I'm not sure that's really better myself)

But at a practical level, we can encourage use of the transient /etc where possible - that makes things work more like a container by default. It should work well in relatively "stateless" cloud deployments. The problem though is, when you go to deploy on bare metal you end up wanting that static /etc/hostname file, your NetworkManager configs...

[stefwalter]

I understand this is a quagmire. So here's my ideal /etc behavior. I am of course less informed than you, but I thought I'd write it down:

1. bootc install and bootc update and bootc switch should all have the same behavior with regards to /etc
2. All of the above should inform me (ie: print out in same way) the list of files they are preserving in /etc and thus ignoring from the Container image.

The above behavior also would fix some of #434

[cgwalters]

BTW a great example of stuff here to think about is the auto-generated SSH host keys, which get built on boot and live in /etc. (I only last year learned that actually Debian's openssh package generates the key in a package-install post-script, which would mean every host generated by a container image has the same keys and the private keys would be in the container image if you do it the naive way. Fedora derivatives do it in a systemd unit)

The mental model here is that bootc install has a fresh /etc and /var, whereas bootc update and bootc switch preserve them.