UniFi OS - Dream Machines 3.x

[pedropombeiro]

Ubiquiti recently made 3.0.19 available in EA at https://community.ui.com/releases/UniFi-OS-Dream-Machines-3-0-19/aae685bb-4b96-4016-9125-29e57d7f2844

Known aspects of 3.x

• /data is preserved in the upgrade, but udm-boot_1.0.7_all.deb needs to be installed again;
• ssh-proxy is not present in 3.x, but manual install runs correctly;
• This release has a kernel which is compiled without CONFIG_BPF_SYSCALL (tracked in https://github.com/unifi-utilities/unifios-utilities/issues/510), meaning podman cannot run on it without a custom kernel;
• ~The /etc/apt/sources.list is still pointing to stretch~. It now points to bullseye.

UPDATE: I've started a discussion at https://github.com/orgs/unifi-utilities/discussions/564 to get some feedback on how the migration to 3.x went for the more adventurous users among us.

[boostchicken]

use udm-boot-2x, it does not use ssh-proxy

[boostchicken]

Did you compain to Ubi regarding apt not pointing at bullseye?

[mabunixda]

Yeah just dropped that info on the UI discussion

[boostchicken]

I havve requested the kernel sources for 3.0.19, lets see how long it takes

[marco3181]

Hello, For the podman problem, there is this message : https://github.com/unifi-utilities/unifios-utilities/issues/510#issuecomment-1478787703 Seems that we can replace it with systemd-nspawn.

[marco3181]

@mabunixda : after your message on ui forum, rekoil says that his sources are pointing at bullseye. Maybe EA problem

[mabunixda]

I'd prefer a common solution that is backwards compatible about podman/docker 🤔

@marco3181 Yeah donot know ... could be EA stuff yeah

[operinko]

My /etc/apt/sources.list is properly pointing to bullseye (and have been since the start). I updated only a few hours after 3.x landed for UDM Pro in EA.

[spali]

Mine too (updates within minutes after EA announcement:

root@udmp:~# cat /etc/apt/sources.list
deb http://deb.debian.org/debian/ bullseye main contrib non-free
deb http://deb.debian.org/debian/ bullseye-updates main contrib non-free
deb http://deb.debian.org/debian/ bullseye-backports main
deb http://security.debian.org/debian-security bullseye-security main contrib non-free

[mabunixda]

Fixed it on my installation - might also be an old EA update that did not change this definitions...

[boostchicken]

See my update here

So I have not upgraded yet, but on 2.5x is overlayfs working, I expect it will work on 3 as well. So the disk space issue is now fixed @peacey

I am pretty sure I can get podman running on the UDMSE. See BPF is used for security lockdown on syscalls. We are always root anyways so it doesnt matter. I am going to drop seccomp and apparmor from the UDMP-UDMSE build and edit configs accordingly and I think that will get you in business.

Also, they new podman builds come with crun, if you want to use it. It's much better on resources. Uses less memory, and executes faster

Also, netavark is built and in the latest zips as well. This is a replacement for CNI which is now deprecated. It has cool things like macvlan dhcp working, but its not zero effort to migrate your networks, syntax of the files is quite different.

I would move to netavark ASAP. It makes containers much much much faster and much less latent on the network. It is written in Rust instead of Go (much like crun). I also included the dhcp-client-proxy if anyone wants to macvlan dhcp working.

I am not sure when I can move to 3. If someone would volunteer to test my new build for 3 build with the mods above I'll crank it out asap that would be awesome.

](https://github.com/unifi-utilities/unifios-utilities/issues/510#issuecomment-1483820487)

[boostchicken]

systemd-nspawn

Yeah that works for the short term, I think I can get podman working see above. It's really just a matter of pulling anything bpf related. Thats all for rootless containers and we don't have to worry about it. Also make sure you do the fix for the disk space when using VFS

[boostchicken]

You know what it just dawned on the current builds should work fine on the latest 3. You just need to edit /usr/share/containers/seccomp.json and disallow any bpf syscalls, BOOM.

Beyound that you can just do this --security-opt=seccomp=unconfined in your podman command and it wont call bpf at all, no security stuff happening.

[boostchicken]

New builds coming out now with a fixxed seccomp.json that removevs bpf from the allows, also fixes some registry issues

[boostchicken]

Grab the two latest builds here if ipfs is being a shit head

https://github.com/unifi-utilities/unifios-utilities/actions

[peacey]

You know what it just dawned on the current builds should work fine on the latest 3. You just need to edit /usr/share/containers/seccomp.json and disallow any bpf syscalls, BOOM.

Beyound that you can just do this --security-opt=seccomp=unconfined in your podman command and it wont call bpf at all, no security stuff happening.

No bueno @boostchicken, it still doesn't work with these modification and your new build. Still same issue with BPF error, unfortunately.

[sidprax]

Yes, same issue and I was wondering if I did something wrong! I also started getting warning for namespace: WARN[0000] Failed to read current user namespace mappings

[ghost]

UDM Boot remote script still works fine. Firmware 3.X comes with DNSCrypt-Proxy service already pre-installed natively, which is a great replacement for simpe local DNS servers and doesn't require Podman!

[georgeboot]

Firmware 3.X comes with DNSCrypt-Proxy service already pre-installed natively, which is a great replacement for simpe local DNS servers and doesn't require Podman!

That's quite interesting! Wondering if I can get that to work with my blocklist (https://oisd.nl). I only use porman for Adguard, so this looks like a nice workaround.

[operinko]

Firmware 3.X comes with DNSCrypt-Proxy service already pre-installed natively, which is a great replacement for simpe local DNS servers and doesn't require Podman!

That's quite interesting! Wondering if I can get that to work with my blocklist (https://oisd.nl). I only use porman for Adguard, so this looks like a nice workaround.

Absolutely can be. https://github.com/DNSCrypt/dnscrypt-proxy/wiki/Public-blocklist & https://github.com/DNSCrypt/dnscrypt-proxy/wiki/Combining-Blocklists

You'll need to trigger the generate-domains-blocklist.py with your configs during cron and possibly reload dnscrypt-proxy afterwards.

[georgeboot]

and possibly reload dnscrypt-proxy afterwards.

From the UDM, i can use it via dig google.com @127.0.2.1 and that works. However, how do I configure this DNS to listen on the UDM's internal IP, instead of dnsmasq? Because currently it's only listening on 127.0.2.1 and thats no routable from my subnets.

[ghost]

I also need specific instructions on how to make DNSCrypt-Proxy the main DNS resolver for UDM for WAN and for LAN.

I think DNSCrypt-Proxy is normally configured via a TOML file, but UDM firmware 3.X uses SystemD, which is listed in DNSCrypt-Proxy manual as a non-standard way of making it work. So... Still need specific instructions.

[paskal]

I've followed @peacey's https://github.com/peacey/unifios-utilities/tree/nspawn/nspawn-container instructions to make nspawn containers replacement work for me. I created an Alpine nspawn container instead of Debian (5Mb instead of 300+, and it takes seconds to bootstrap) using https://gist.github.com/sfan5/52aa53f5dca06ac3af30455b203d3404#file-alpine-container-sh with replacing x86 in text with aarch64. I didn't do any setup with passwd and so on, the container just works.

With installing the multicast package from the community repo within alpine, I've got multicast working for my Sonos in a separate VLAN with such nspawn config for the container (called /etc/systemd/nspawn/alpine-multicast.nspawn in my case):

[Exec]
Boot=on
Capability=all
Parameters=multicast-relay.py --interfaces br0 br4 --foreground

[Network]
Private=off
VirtualEthernet=off
ResolvConf=off

[gatesry]

@paskal - An on boot script also works to just call the python script directly.

[Stealthii]

@gatesry do you have an example?

[gatesry]

@Stealthii For sure!

#!/bin/sh

/usr/bin/python3 /data/custom/multicast-relay/multicast-relay.py --interfaces br0 br20 br30

[Stealthii]

Thanks @gatesry! I'd no idea that the script was that simple all the dependencies were available in the UDM environment. No real need for a container.

[ghost]

Is it possible to make the on-boot script load sooner? I think it is set to "After/Wants=network-online.target". It loads my scripts like custom IPTables after UDM makes network connections. I need my custom rules to load before such connections happen. With old UniFi-OS 1.X firmware, booting sequence was stricter, I think.

FYI, there is no more need for SSH on-boot scripts. Once authorized_keys file is made, it persists after reboots on its own.

[paskal]

@gatesry thanks for the idea, actually I am not using on-boot now already, and with your suggestion, I can use systemd and remove nspawn usage.

[gabbeltje]

@GY8VSdYYzvL8-K6T

Do you have an example for an systemd scipt? I have created one but the script is nog starting.

This is my systemd service file:

[Unit] Wants=network-online.target After=network-online.target

[Service] ExecStart=/data/scripts/multicast-relay/multicast-relay.sh

[Install] WantedBy=multi-user.target

The WantedBy i already tried network-online.target but that is also not working. The paths to the script are correct and when i run the multicast-relay.sh manually it is working.

This is the output of systemctl status multicast.service:

multicast.service Loaded: loaded (/etc/systemd/system/multicast.service; enabled; vendor preset: enabled) Active: inactive (dead) since Mon 2023-04-03 11:53:39 CEST; 1min 45s ago Main PID: 2471 (code=exited, status=0/SUCCESS) CPU: 169ms

Apr 03 11:53:39 UDMPro systemd[1]: Started multicast.service. Apr 03 11:53:39 UDMPro systemd[1]: multicast.service: Succeeded.