Finalizing staged deployments broken on /boot automount

[dbnicholson]

Recently we changed our updater to use staged deployments in endlessm/eos-updater#298. That worked fine on systems where /boot is a persistent mount point, but it fails on systems that use systemd-boot where /boot is the automounted EFI system partition. There are 2 problems with this:

1. If the /boot automount expires, the ostree-finalize-staged.service unit runs immediately since it has RequiresMountsFor=/boot. With nothing keeping the automount from expiring, this can happen at any point prior to shutdown and ruin the feature. This actually deadlocks in systemd, but it would be bad even without the automounting bugs.
2. If RequiresMountsFor=/boot is removed and instead just After=boot.mount is used, then the service is only triggered on shutdown but the ordering remains. However, if the automount has expired, systemd will ignore the request to remount it since the automount is scheduled to be stopped.

See systemd/systemd#22528 for details. Maybe the solution here is staged deployments are not supported on /boot automounts, but I wanted to open for discussion.

[dbnicholson]

The dirty idea I had was to change ostree admin finalize-staged so it opens /boot (and /sysroot) when starting and then blocks waiting for SIGTERM via sigwait. This would effectively turn /boot into a persistent mount if an deployment was scheduled for finalizing since the automount would never expire.

[cgwalters]

I think that approach isn't dirty at all - it makes sense to me. The code is already heavily oriented towards using directory file descriptors, so we already have a natural mechanism to hold open the mounts.

[cgwalters]

And that would actually change us from using ExecStop= to ExecStart= which is definitely more natural.

[dbnicholson]

Alright, I'll put something together. One question I had is, what if someone unstages the deployment? Should it watch /run/ostree/staged-deployment for deletion and stop itself without finalizing? I think finalizing is pretty much idempotent, so probably it could be allowed to run even if someone unstages.

However, you would need to change the builtin to not initially lock the sysroot since that would prevent doing anything else with it until the unit was stopped. So, I think you'd want to block on the signal, receive the signal, lock the sysroot, load it again (so the state of the deployments is up to date), and then finalize.

[cgwalters]

Should it watch /run/ostree/staged-deployment for deletion and stop itself without finalizing? I think finalizing is pretty much idempotent, so probably it could be allowed to run even if someone unstages.

If you want to handle this case, that sounds good to me, but it doesn't seem at all required to me. It's a real corner case, and we aren't going to be holding open much resident memory. And we can fix it later if someone actually does complain, so probably keep it simple to start.

Plus, people should be applying kernel updates and rebooting anyways :smile: