almalinux/rockylinux 9 network failure due to AppArmor denying nm-dhcp-helper

[dimitry-unified-streaming]

Running almalinux 9 or rockylinux 9 lxc containers (unprivileged) will result in no networking being available in those containers. Even taking lxc/lxc#2778 and lxc/linuxcontainers.org/issues/580 into account, e.g. specifying lxc.apparmor.profile = unconfined in the lxc configuration file, there is something in AppArmor that causes the containers to fail to start nm-dhcp-helper.

I.e. on a Ubuntu 20.04 LTS server with lxc 1:4.0.12-0ubuntu1~20.04.1 installed:

{ echo "lxc.include = /etc/lxc/default.conf"; echo "lxc.apparmor.profile = unconfined"; echo "lxc.idmap = u 0 100000 65536"; echo "lxc.idmap = g 0 100000 65536"; } > rocky9.conf
lxc-create -n rocky9 -f rocky9.conf -t download -- -d rockylinux -r 9 -a amd64
lxc-start -n rocky9

results in the container starting, but if you attach with lxc-attach -n rocky9 you see in journalctl output:

...
Aug 10 14:53:19 rocky9 NetworkManager[104]: <info>  [1660143199.6201] device (eth0): Activation: starting connection 'System eth0' (5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03)
Aug 10 14:53:19 rocky9 NetworkManager[104]: <info>  [1660143199.6202] device (eth0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Aug 10 14:53:19 rocky9 NetworkManager[104]: <info>  [1660143199.6203] manager: NetworkManager state is now CONNECTING
Aug 10 14:53:19 rocky9 NetworkManager[104]: <info>  [1660143199.6204] device (eth0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Aug 10 14:53:19 rocky9 NetworkManager[104]: <info>  [1660143199.6208] device (eth0): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Aug 10 14:53:19 rocky9 NetworkManager[104]: <info>  [1660143199.6211] dhcp4 (eth0): activation: beginning transaction (timeout in 45 seconds)
Aug 10 14:53:19 rocky9 NetworkManager[104]: <info>  [1660143199.6227] dhcp4 (eth0): dhclient started with pid 121
Aug 10 14:53:19 rocky9 systemd[1]: Started Network Manager Script Dispatcher Service.
Aug 10 14:53:19 rocky9 dhclient[125]: execve (/usr/libexec/nm-dhcp-helper, ...): Permission denied
Aug 10 14:53:19 rocky9 dhclient[121]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 6 (xid=0x73883c22)
Aug 10 14:53:22 rocky9 dhclient[121]: DHCPOFFER of 10.0.3.151 from 10.0.3.1
Aug 10 14:53:22 rocky9 dhclient[121]: DHCPREQUEST for 10.0.3.151 on eth0 to 255.255.255.255 port 67 (xid=0x73883c22)
Aug 10 14:53:22 rocky9 dhclient[121]: DHCPACK of 10.0.3.151 from 10.0.3.1 (xid=0x73883c22)
Aug 10 14:53:22 rocky9 dhclient[138]: execve (/usr/libexec/nm-dhcp-helper, ...): Permission denied
Aug 10 14:53:22 rocky9 dhclient[121]: bound to 10.0.3.151 -- renewal in 1503 seconds.

Even though dhclient says "bound to 10.0.3.151", the previous execve (/usr/libexec/nm-dhcp-helper, ...): Permission denied line causes the container to have no IP address on its eth0 interface:

[root@rocky9 /]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:78:c7:80 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::216:3eff:fe78:c780/64 scope link
       valid_lft forever preferred_lft forever

It turns out that AppArmor on the Ubuntu host system has a rule for nm-dhcp-helper, but it's in a different path than Red Hat's (because different distros are, well, different):

$ cat /etc/apparmor.d/sbin.dhclient
...
  # The dhclient-script shell script sources other shell scripts rather than
  # executing them, so we can't just use a separate profile for dhclient-script
  # with 'Uxr' on the hook scripts. However, for the long-running dhclient3
  # daemon to run arbitrary code via /sbin/dhclient-script, it would need to be
  # able to subvert dhclient-script or write to the hooks.d directories. As
  # such, if the dhclient3 daemon is subverted, this effectively limits it to
  # only being able to run the hooks scripts.
  /{,usr/}sbin/dhclient-script                           Uxr,

  # Run the ELF executables under their own unrestricted profiles
  /usr/lib/NetworkManager/nm-dhcp-client.action   Pxrm,
  /usr/lib/connman/scripts/dhclient-script        Pxrm,

  # Support the new executable helper from NetworkManager.
  /usr/lib/NetworkManager/nm-dhcp-helper          Pxrm,
  signal (receive) peer=/usr/lib/NetworkManager/nm-dhcp-helper,

  # Site-specific additions and overrides. See local/README for details.
  #include <local/sbin.dhclient>
}
...
/usr/lib/NetworkManager/nm-dhcp-helper {
  #include <abstractions/base>
  #include <abstractions/dbus>
  /usr/lib/NetworkManager/nm-dhcp-helper mr,

  /run/NetworkManager/private-dhcp rw,
  signal (send) peer=/sbin/dhclient,

  /var/lib/NetworkManager/*lease r,
  signal (receive) peer=/usr/sbin/NetworkManager,
  ptrace (readby) peer=/usr/sbin/NetworkManager,
  network inet dgram,
  network inet6 dgram,
}

I.e. for some reason, even if lxc.apparmor.profile is apparently 'unconfined', AppArmor still denies the execution of /usr/libexec/nm-dhcp-helper, because it's path isn't /usr/lib/NetworkManager/nm-dhcp-helper !

Aug 10 16:55:34 ubu2004 kernel: [ 4694.047779] audit: type=1400 audit(1660143334.650:62): apparmor="DENIED" operation="exec" profile="/{,usr/}sbin/dhclient" name="/usr/libexec/nm-dhcp-helper" pid=25671 comm="dhclient" requested_mask="x" denied_mask="x" fsuid=100000 ouid=100000

I have verified that if I completely disable AppArmor on the host, the problem goes away, and also if I edit /etc/apparmor.d/sbin.dhclient and add a copy of the nm-dhcp-helper parts with the path changed to the Red Hat location.

But isn't there a way to fix this from the distrobuilder side? I.e. some fix to be included in the Rockylinux and Almalinux container images?

[dimitry-unified-streaming]

FWIW the hack I did on the Ubuntu host side's AppArmor configuration is:

diff --git a/apparmor.d/sbin.dhclient b/apparmor.d/sbin.dhclient
index 1acc6b9..2725f1d 100644
--- a/apparmor.d/sbin.dhclient
+++ b/apparmor.d/sbin.dhclient
@@ -67,8 +67,8 @@
   /usr/lib/connman/scripts/dhclient-script        Pxrm,
 
   # Support the new executable helper from NetworkManager.
-  /usr/lib/NetworkManager/nm-dhcp-helper          Pxrm,
-  signal (receive) peer=/usr/lib/NetworkManager/nm-dhcp-helper,
+  /usr/{lib/NetworkManager,libexec}/nm-dhcp-helper          Pxrm,
+  signal (receive) peer=/usr/{lib/NetworkManager,libexec}/nm-dhcp-helper,
 
   # Site-specific additions and overrides. See local/README for details.
   #include <local/sbin.dhclient>
@@ -86,10 +86,10 @@
   network inet6 dgram,
 }
 
-/usr/lib/NetworkManager/nm-dhcp-helper {
+/usr/{lib/NetworkManager,libexec}/nm-dhcp-helper {
   #include <abstractions/base>
   #include <abstractions/dbus>
-  /usr/lib/NetworkManager/nm-dhcp-helper mr,
+  /usr/{lib/NetworkManager,libexec}/nm-dhcp-helper mr,
 
   /run/NetworkManager/private-dhcp rw,
   signal (send) peer=/sbin/dhclient,

[monstermunchkin]

Distrobuilder doesn't mess with AppArmor as it's only an image builder. This issue needs to be handled on the host system.

Another workaround could be to build your own image, and moving nm-dhcp-helper to the appropriate path using a post-files hook.