Container issues with Ubuntu Oracular systemd version 256-1ubuntu1

[mihalicyn]

Investigate and fix all the issues

TODO list:

• [x] fix unprivileged containers case (covered by https://github.com/canonical/lxd/pull/13820)
• [ ] try to fix privileged containers case (we need https://github.com/canonical/lxd-imagebuilder/commit/c928d22574893208fd9ddbdb8c7dcf7e3899c19d )
• [ ] land necessary fixes in LXC for both cases (LXC ships it's own AppArmor profiles) (covered by https://github.com/lxc/lxc/pull/4466)
• [ ] communicate with AppArmor team about possible bugs & land necessary fixes to upstream ( https://lore.kernel.org/all/[email protected]/ )

See also: https://github.com/canonical/lxd/issues/13807 https://github.com/canonical/lxd/pull/12698 https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2046486 https://github.com/canonical/lxd/issues/13844 https://github.com/canonical/lxd/issues/13860

[tomponline]

For now the workaround is to do lxc config set <instance> security.nesting=true and restart container.

[tomponline]

Reopens for privilege use case.

[mihalicyn]

Ok, what we have about privileged containers:

• jammy works
• noble doesn't
• oracular too

What we have in the audit logs:

[15669.318153] audit: type=1400 audit(1721991292.665:4813): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-no_</var/snap/lxd/common/lxd>" name="/run/systemd/mount-rootfs/" pid=145591 comm="(d-logind)" srcname="/" flags="rw, rbind"
[15669.327424] audit: type=1400 audit(1721991292.673:4814): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-no_</var/snap/lxd/common/lxd>" name="/run/systemd/mount-rootfs/" pid=145595 comm="(d-logind)" srcname="/" flags="rw, rbind"
[15669.336943] audit: type=1400 audit(1721991292.685:4815): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-no_</var/snap/lxd/common/lxd>" name="/run/systemd/mount-rootfs/" pid=145599 comm="(d-logind)" srcname="/" flags="rw, rbind"
[15669.346912] audit: type=1400 audit(1721991292.693:4816): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-no_</var/snap/lxd/common/lxd>" name="/run/systemd/mount-rootfs/" pid=145603 comm="(d-logind)" srcname="/" flags="rw, rbind"
[15669.357968] audit: type=1400 audit(1721991292.705:4817): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-no_</var/snap/lxd/common/lxd>" name="/run/systemd/mount-rootfs/" pid=145607 comm="(d-logind)" srcname="/" flags="rw, rbind"
[15671.673932] audit: type=1400 audit(1721991295.021:4818): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-or_</var/snap/lxd/common/lxd>" name="/run/systemd/mount-rootfs/" pid=145649 comm="(d-logind)" srcname="/" flags="rw, rbind"
[15671.674044] audit: type=1400 audit(1721991295.021:4819): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-or_</var/snap/lxd/common/lxd>" name="/dev/" pid=145651 comm="(sd-mkdcreds)" flags="rw, rslave"
[15671.683716] audit: type=1400 audit(1721991295.029:4820): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-or_</var/snap/lxd/common/lxd>" name="/run/systemd/mount-rootfs/" pid=145654 comm="(d-logind)" srcname="/" flags="rw, rbind"
[15671.686683] audit: type=1400 audit(1721991295.033:4821): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-or_</var/snap/lxd/common/lxd>" name="/dev/" pid=145658 comm="(sd-mkdcreds)" flags="rw, rslave"
[15671.692154] audit: type=1400 audit(1721991295.037:4822): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-or_</var/snap/lxd/common/lxd>" name="/run/systemd/mount-rootfs/" pid=145659 comm="(d-logind)" srcname="/" flags="rw, rbind"

This thing wants a recursive bind-mount of / to /run/systemd/mount-rootfs/. While it's perfectly safe for unprivileged containers, we can't allow it for privileged ones. It would be a security breach.

We have this commit in our imagebuilder https://github.com/canonical/lxd-imagebuilder/commit/c928d22574893208fd9ddbdb8c7dcf7e3899c19d and taking into account that images:debian/sid container works with security.privileged=true I would just apply the same quirk to the Ubuntu images.

[tomponline]

Have asked CPC about being able to apply a similar fix as https://github.com/canonical/lxd-imagebuilder/commit/c928d22574893208fd9ddbdb8c7dcf7e3899c19d to ubuntu: images.

https://chat.canonical.com/canonical/pl/dygqosfbp3gy3x7hcs5ssergoy

[tomponline]

@mihalicyn does this fix rely on the apparmor parser version in core24?

[mihalicyn]

@mihalicyn does this fix rely on the apparmor parser version in core24?

yes, but we don't need core24 itself.

Full list of changes we need to backport:

1. https://github.com/canonical/lxd-pkg-snap/pull/477
2. https://github.com/canonical/lxd/pull/13681
3. https://github.com/canonical/lxd/pull/13820 (optional) https://github.com/canonical/lxd/pull/13826

[tomponline]

thanks @mihalicyn

I've added the vendored apparmor parser and your fixes as cherry-picks to latest/stable now.

[tomponline]

We are rolling out fixes to 5.21/stable for this now (snap refresh lxd --channel=5.21/stable --cohort="+" will get it by bypassing progressive rollout).

[tomponline]

@mihalicyn please can you verify that Oracular is working on Jammy host with 5.0/edge channel, if so then we can close this as the fix will be included in LXD 5.0.4 Thanks

[mihalicyn]

I can confirm that it works for me well on 5.0/edge with Ubuntu Jammy host.

[tomponline]

LXD 5.0.4 now supports launching Oracular containers on a cgroupv2 enabled host

https://discourse.ubuntu.com/t/lxd-5-0-4-lts-has-been-released/49681