remove $stateroot/var rw bind mount / improve do_remount()

[champtar]

- ostree-remount: improve do_remount(), use warn()

When we remount read only, we only want to change the vfs read only
flag, not the fs one, so use MS_BIND.
On the contrary when we remount read write, we want to change both (already ok).
To check that we are allowed to write, we now use 'access'.

We should never need do_remount(), as everything should have been
properly setup by prepare-root and ostree-system-generator,
so start to warn() when we actually remounted something.

- prepare-root: remove $stateroot/var rw bind mount

Now that we have mount.ostreebindrw helper, we can safely
remove the $stateroot/var rw bind mount.

This allow to have /sysroot fully read-only.

- generator: add mount.ostreebindrw helper

Recent versions of mount that use the new mount API (util-linux 2.39)
will honor the 'bind,rw' mount option and create a writable bind mount
even if the source is read-only.
For older mount versions, install a simple mount helper that create
the bind mount then remount bind,rw it.
This avoids having /var read-only before ostree-remount runs.

- prepare-root/generator: drop /run/ostree/initramfs-mount-var support

Ignition stopped using it in 2019 (https://github.com/coreos/ignition/commit/852008ffcc205c15e6b6a0d1b2deaf33fb2bad79),
and it has been broken for 2 years (https://github.com/ostreedev/ostree/commit/64afbcdeb0cb101ce302ae28f1c0852a9d73b907)
(mkdirat(/run/ostree) fails if /run/ostree already exists)

Note: This requires a small change on CoreOS side: https://github.com/coreos/fedora-coreos-config/pull/3773

Fixes #3519

[openshift-ci[bot]]

Skipping CI for Draft Pull Request. If you want CI signal for your change, please convert it to an actual PR. You can still manually trigger a test run with /test all

[champtar]

Blocked by https://github.com/coreos/fedora-coreos-config/pull/3773

[jlebon]

Would be good to sanity-check el9-based SCOS before merging this.

[champtar]

/test all

[champtar]

Would be good to sanity-check el9-based SCOS before merging this.

Done, it boots fine

[core@cosa-devsh ~]$ journalctl -u ostree-remount.service 
Sep 23 15:37:17 localhost systemd[1]: Starting OSTree Remount OS/ Bind Mounts...
Sep 23 15:37:17 localhost ostree-remount[1543]: Remounted rw: /var
Sep 23 15:37:17 localhost systemd[1]: Finished OSTree Remount OS/ Bind Mounts.

[core@cosa-devsh ~]$ grep '/sysroot' /proc/1/mountinfo 
75 1 0:33 / / ro,relatime shared:1 - overlay composefs ro,seclabel,lowerdir+=/run/ostree/.private/cfsroot-lower,datadir+=/sysroot/ostree/repo/objects,redirect_dir=on,metacopy=on
76 75 252:4 / /sysroot ro,relatime shared:3 - xfs /dev/vda4 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota

[core@cosa-devsh ~]$ systemctl cat var.mount 
# /run/systemd/generator/var.mount
##
# Automatically generated by ostree-system-generator
##

[Unit]
Documentation=man:ostree(1)
ConditionKernelCommandLine=!systemd.volatile
Before=local-fs.target

[Mount]
Where=/var
What=/sysroot/ostree/deploy/scos/var
Options=bind,rw,slave,shared

[champtar]

I've now added a small mount helper so var.mount is writable right away.