Openwrt procd jail

[M1cha]

First, thx for the openwrt images, they're very useful :) The upcoming release will introduce proc jails for process isolation. The current openwrt.yaml for distrobuilder already disables the jail for dnsmasq as a workaround. For the latest snapshots that's not enough anymore since sysntpd now uses jails too and I guess more services will do so in future.

While disabling them all would be an easy way out, it'd obviously be better to support the jail inside LXD to enjoy better security within the container. So the question becomes: what's needed to do that? Simply enable nesting? Would nesting have other security implications that may not be worth it?

[monstermunchkin]

I just built an OpenWrt 22.03 image, and sysntpd works without having to disable the proc jail. For dnsmasq however it still seems necessary to disable it, but I don't know why. Enabling security.nesting doesn't change the behaviour.

[ntaken]

I just built an OpenWrt 22.03 image, and sysntpd works without having to disable the proc jail. For dnsmasq however it still seems necessary to disable it, but I don't know why. Enabling security.nesting doesn't change the behaviour.

Interesting does it still work?

When I try with the 22.03 arm64 image from https://us.lxd.images.canonical.com/ I get "daemon.info procd: Instance sysntpd::instance1 s in a crash loop". But perhaps that is a different problem.

[simondeziel]

I just tested with images:openwrt/22.03 (d5e15fba1479 for amd64) and I also get daemon.info procd: Instance sysntpd::instance1 s in a crash loop 6 crashes, 1 seconds since last crash but only ~30-45 seconds after booting the container.

I also found that setting security.nesting=true avoids many jail errors. logread looks much cleaner with it at least.

[ghost]

Fri Apr 28 06:28:54 2023 user.err : jail: failed to mount /var/run/ubus/ubus.sock /tmp/ujail-daeBpN/var/run/ubus/ubus.sock: Permission denied
Fri Apr 28 06:28:54 2023 user.err : jail: mount_all() failed
Fri Apr 28 06:28:54 2023 user.err : jail: failed to build jail fs
Fri Apr 28 06:28:59 2023 user.err : jail: failed to mount /var/run/ubus/ubus.sock /tmp/ujail-IifFlI/var/run/ubus/ubus.sock: Permission denied
Fri Apr 28 06:28:59 2023 user.err : jail: mount_all() failed
Fri Apr 28 06:28:59 2023 user.err : jail: failed to build jail fs
Fri Apr 28 06:29:04 2023 user.err : jail: failed to mount /var/run/ubus/ubus.sock /tmp/ujail-ofLPGK/var/run/ubus/ubus.sock: Permission denied
Fri Apr 28 06:29:04 2023 user.err : jail: mount_all() failed
Fri Apr 28 06:29:04 2023 user.err : jail: failed to build jail fs
Fri Apr 28 06:29:09 2023 user.err : jail: failed to mount /var/run/ubus/ubus.sock /tmp/ujail-Fnamag/var/run/ubus/ubus.sock: Permission denied
Fri Apr 28 06:29:09 2023 user.err : jail: mount_all() failed
Fri Apr 28 06:29:09 2023 user.err : jail: failed to build jail fs
Fri Apr 28 06:29:14 2023 user.err : jail: failed to mount /var/run/ubus/ubus.sock /tmp/ujail-OmhBhf/var/run/ubus/ubus.sock: Permission denied
Fri Apr 28 06:29:14 2023 user.err : jail: mount_all() failed
Fri Apr 28 06:29:14 2023 user.err : jail: failed to build jail fs
Fri Apr 28 06:29:14 2023 daemon.info procd: Instance sysntpd::instance1 s in a crash loop 6 crashes, 0 seconds since last crash

I have the same problem with lxc running latest openwrt 22.03 on pve.

[pbix]

Same messages here as reported by jukkui.

sysntpd is in a crash loop and not functioning as a result.

[stgraber]

Is that still an issue with the more recent OpenWRT images?

[tew42]

I did encounter this with 22.03 in the past month, but now on 23.05 (with features: keyctl=1,nesting=1) it seems fine

[stgraber]

Okay, that's good to hear. Should we do something about the earlier releases or do you expect folks will just care about 23.05 anyway?

[tew42]

Actually never mind, seems I spoke too soon. Not entirely sure why I didn't see it in the initial instance, but verified now both after reboot of fully-configured instance, and a brand-new LXC:

Wed Oct 18 21:57:16 2023 daemon.info procd: Instance sysntpd::instance1 s in a crash loop 6 crashes, 1 seconds since last crash Wed Oct 18 22:10:44 2023 daemon.info procd: Instance sysntpd::instance1 s in a crash loop 6 crashes, 1 seconds since last crash

[stgraber]

Should we do the same trick as dnsmasq then?

[tew42]

Just tried implementing the same "sed" as for dnsmasq on init.d/sysntpd, but this turns out to not be sufficient, the error persists. I suspect it has to do with the following piece of code between procd_set_param respawn and all the procd_add_jail calls which diverges distinctly from the dnsmasq implementation. Further troubleshooting is beyond my capabilities on this, unfortunately:

[ -x /sbin/ujail -a -e /etc/capabilities/ntpd.json ]

Addendum: One potential "fix" could be to just disable the sysntpd process/service altogether, since in a container we could assume the host to be responsible for system time provision anyway?

[stgraber]

Oh indeed, that'd be the better fix as ntpd cannot work in a container anyway.

[M1cha]

Just out of curiosity, why can't ntpd work in a container?

[stgraber]

because ntpd needs to be able to alter the system clock to function, as that clock is global on the entire system, individual containers aren't allowed to mess with it.

[daiaji]

because ntpd needs to be able to alter the system clock to function, as that clock is global on the entire system, individual containers aren't allowed to mess with it.

In fact, bare metal OS may have many ways to make NTP work.