udm-kernel-tools
udm-kernel-tools copied to clipboard
fabianishere
Support for UDM SE
Hi,
I wanted to install a custom kernel for the UDMSE (version 2.5.11), however, it seems to fail:
root@FW:~# uname -a
Linux FW 4.19.152-ui-alpine #4.19.152 SMP Mon Aug 1 14:24:56 CST 2022 aarch64 GNU/Linux
root@FW:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@FW:~# udm-bootctl list
Version Initrd
4.19.152-edge4 yes
root@FW:~# udm-bootctl boot 4.19.152-edge4
/usr/bin/udm-bootctl: line 193: ssh-proxy: command not found
I assume this is due to the new 'major' UniFi-OS version, as the UDM and UDMP are at 1.12.30, while the UDMSE and UDR are at 2.5.11. However, I was unable to find any documentation on this - is there any?
Thank you!
You are correct, the UDM SE and UDR are currently unsupported since they were released after this project was developed.
Porting this project to those devices requires building the kexec-mod project for the kernels running on these devices. However, I have no way of testing this since I don’t own a UDM SE.
Once the UDM (Pro) is on 2.x, I will start porting this project to the new firmware.
If you only want wireguard support this project works great on UDM-SE:
https://github.com/tusc/wireguard-kmod
Hi @fabianishere wondering if this is still on your radar? :) (We now have moved to Unifi OS 3.x)
This is still on my ToDo list, but I don't have access to a UDM Pro at the moment. I'll probably start working on this start of next year.
UDM/Pro is being updated to the same 3.x branch with UDM Pro SE 3.x, migration is already started for early access so this issue will probably apply to UDM/Pro soon once they go GA
@fabianishere: I have got an udm se and am interested to help getting this project running. Could you point out what we have to achive to get it running? Some advise how to start?
@fabianishere: I have got an udm se and am interested to help getting this project running. Could you point out what we have to achive to get it running? Some advise how to start?
You'll have to retrieve the kernel sources from Ubiquiti and verify whether you can build kexec-mod for the kernel.
Then it is a matter of updating the udm-bootctl command to not make use of ssh-proxy, but invoke the commands on the system directly.
Is there an specific adress at unifi to get the sources? Who did you turn to to get the sources?
Am 21.01.2023 um 21:06 schrieb Fabian Mastenbroek:
@fabianishere <https://github.com/fabianishere>: I have got an udm se and am interested to help getting this project running. Could you point out what we have to achive to get it running? Some advise how to start?You'll have to retrieve the kernel sources from Ubiquiti and verify whether you can build kexec-mod https://github.com/fabianishere/kexec-mod for the kernel.
Then it is a matter of updating the udm-bootctl https://github.com/fabianishere/udm-kernel-tools/blob/master/udm-bootctl command to not make use of |ssh-proxy|, but invoke the commands on the system directly.
— Reply to this email directly, view it on GitHub https://github.com/fabianishere/udm-kernel-tools/issues/82#issuecomment-1399323544, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACYGU3HAXJI6U6Y7RVSLELWTQ6VPANCNFSM6AAAAAARCMKEGA. You are receiving this because you commented.Message ID: @.***>
I have got the kernel sources and will take a look wether i can build kexec-mod.
ok i get the following error:
make KDIR=../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine
make -C ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine M=/mnt/d/github/kexec-mod/kernel modules
make[1]: Entering directory '/mnt/d/github/kexec-mod/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine'
mkdir -p /mnt/d/github/kexec-mod/kernel/orig
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec.c /mnt/d/github/kexec-mod/kernel/orig/kexec.c
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec_core.c /mnt/d/github/kexec-mod/kernel/orig/kexec_core.c
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec_internal.h /mnt/d/github/kexec-mod/kernel/orig/kexec_internal.h
scripts/Makefile.build:45: /mnt/d/github/kexec-mod/kernel/arch/x86/Makefile: No such file or directory
make[3]: *** No rule to make target '/mnt/d/github/kexec-mod/kernel/arch/x86/Makefile'. Stop.
make[2]: *** [scripts/Makefile.build:544: /mnt/d/github/kexec-mod/kernel/arch/x86] Error 2
make[1]: *** [Makefile:1550: _module_/mnt/d/github/kexec-mod/kernel] Error 2
make[1]: Leaving directory '/mnt/d/github/kexec-mod/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine'
make: *** [Makefile:15: module] Error 2
Did you build kexec-mod on the udm? I tried using the windows subsystem for linux. I will try using a plain linux later.
ok i get the following error:
make KDIR=../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine make -C ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine M=/mnt/d/github/kexec-mod/kernel modules make[1]: Entering directory '/mnt/d/github/kexec-mod/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine' mkdir -p /mnt/d/github/kexec-mod/kernel/orig ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec.c /mnt/d/github/kexec-mod/kernel/orig/kexec.c ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec_core.c /mnt/d/github/kexec-mod/kernel/orig/kexec_core.c ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec_internal.h /mnt/d/github/kexec-mod/kernel/orig/kexec_internal.h scripts/Makefile.build:45: /mnt/d/github/kexec-mod/kernel/arch/x86/Makefile: No such file or directory make[3]: *** No rule to make target '/mnt/d/github/kexec-mod/kernel/arch/x86/Makefile'. Stop. make[2]: *** [scripts/Makefile.build:544: /mnt/d/github/kexec-mod/kernel/arch/x86] Error 2 make[1]: *** [Makefile:1550: _module_/mnt/d/github/kexec-mod/kernel] Error 2 make[1]: Leaving directory '/mnt/d/github/kexec-mod/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine' make: *** [Makefile:15: module] Error 2Did you build kexec-mod on the udm? I tried using the windows subsystem for linux. I will try using a plain linux later.
You need to cross-compile to ARM64, by setting ARCH=arm64 and CROSS_COMPILE=aarch64-linux-gnu- (make sure you have gcc-aarch64-linux-gnu installed)
I cam here to share I got sources but I have been defeated!
Well done! Its quite telling the UDMSE sources came quick but UDMPRO, nada!
With cross compile i get a different error. Any hint for this?
export ARCH=arm64; export CROSS_COMPILE=aarch64-linux-gnu-; make KDIR=../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine
make -C ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine M=/mnt/d/github/kexec-mod/kernel modules
make[1]: Entering directory '/mnt/d/github/kexec-mod/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine'
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec.c /mnt/d/github/kexec-mod/kernel/orig/kexec.c
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec_core.c /mnt/d/github/kexec-mod/kernel/orig/kexec_core.c
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec_internal.h /mnt/d/github/kexec-mod/kernel/orig/kexec_internal.h
mkdir -p /mnt/d/github/kexec-mod/kernel/arch/arm64/orig
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/arch/arm64/kernel/machine_kexec.c /mnt/d/github/kexec-mod/kernel/arch/arm64/orig/machine_kexec.c
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/arch/arm64/kernel/relocate_kernel.S /mnt/d/github/kexec-mod/kernel/arch/arm64/orig/relocate_kernel.S
CC [M] /mnt/d/github/kexec-mod/kernel/arch/arm64/machine_kexec_drv.o
CC [M] /mnt/d/github/kexec-mod/kernel/arch/arm64/machine_kexec_compat.o
CC [M] /mnt/d/github/kexec-mod/kernel/arch/arm64/idmap.o
AS [M] /mnt/d/github/kexec-mod/kernel/arch/arm64/cpu-reset.o
AS [M] /mnt/d/github/kexec-mod/kernel/arch/arm64/hyp-shim.o
CC [M] /mnt/d/github/kexec-mod/kernel/arch/arm64/machine_kexec.o
/mnt/d/github/kexec-mod/kernel/arch/arm64/machine_kexec.c:2:10: fatal error: orig/machine_kexec.c: No such file or directory
2 | #include "orig/machine_kexec.c"
| ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[3]: *** [scripts/Makefile.build:304: /mnt/d/github/kexec-mod/kernel/arch/arm64/machine_kexec.o] Error 1
make[2]: *** [scripts/Makefile.build:544: /mnt/d/github/kexec-mod/kernel/arch/arm64] Error 2
make[1]: *** [Makefile:1550: _module_/mnt/d/github/kexec-mod/kernel] Error 2
make[1]: Leaving directory '/mnt/d/github/kexec-mod/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine'
make: *** [Makefile:15: module] Error 2
Ok got it working by using the absolut path for KDIR not a relative one. Build finished. Make of the User-space helper was also successfull. What are the next steps?
Build a custom kerneland make sure to add the configs for BPF needed for podman. Then you can follow the docs and boot into that kernel.
Ok i am just building the kernel. How can i test it manually? Just by coping the kernel, kexex-module and redir.so to the udm and then run LD_PRELOAD=/root/redir.so kexec -l /path/to/custom/kernel --reuse-cmdline?
@foxcris just wanted to reach out and see if you were able to successfully boot into this kernel on your SE and run containers?
Today i had some time to go on with kexec an a custom kernel. Currently i still didn't manage to boot a custom kernel. My problem is that i cannot load the kexec_mod module. First isse was the wrong version magic
kexec_mod: version magic '4.19.152+ SMP mod_unload aarch64' should be '4.19.152-ui-alpine SMP mod_unload aarch64'
I figured out how to fix this by setting the EXTRAVERSION. However it took me a while to "remove" the "+" sign. Wasn't really obvious and caused by git (you have to set LOCALVERSION manually during build). Now the version magic missmatch is gone but i get different errors when i try to load the module with modprobe.
kexec_mod: Unknown symbol machine_kexec_prepare (err -2)
kexec_mod: Unknown symbol machine_crash_shutdown (err -2)
kexec_mod: Unknown symbol machine_kexec (err -2)
kexec_mod: Unknown symbol machine_kexec_cleanup (err -2)
@fabianishere: Any hints how to go on now? As soon as i can manually boot the custom kernel i will try to build/adapt your packages.
Oh i just missed to load "kexec_mod_arm64.ko" first. Kexec_mod is now loaded. I will try to boot a custom kernel as a next step.
Kexec is still compaining about miss /proc/kcore.
root@UDM-SE:/ssd1/custom_data# LD_PRELOAD=/ssd1/custom_data/user/redir.so kexec -l /ssd1/custom_data/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine/vmlinux --reuse-cmdline
ERROR: ld.so: object '/ssd1/custom_data/user/redir.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
Can't open (/proc/kcore).
Can't open (/proc/kcore).
kexec_load failed: Function not implemented
entry = 0xcac690 flags = 0xb70000
nr_segments = 4
segment[0].buf = 0x7f82b5a010
segment[0].bufsz = 0x8415f0
segment[0].mem = 0x80000
segment[0].memsz = 0x842000
segment[1].buf = 0x7f833aa010
segment[1].bufsz = 0x380200
segment[1].mem = 0x8d0000
segment[1].memsz = 0x3cf000
segment[2].buf = 0x556552b060
segment[2].bufsz = 0x68e7
segment[2].mem = 0xca5000
segment[2].memsz = 0x7000
segment[3].buf = 0x55655327a0
segment[3].bufsz = 0x3460
segment[3].mem = 0xcac000
segment[3].memsz = 0x4000
Ok found the error. I used the wrong path to redir.so. Now the kernel is loaded. Will try to boot it with kexec -e.
After running
kexec -e
they system is hanning. The display on the router tells something about plugging in the wan cable (but its still plugged in). I needed to reboot it manually by switching it off and on again. With
systemctl kexec
the system directly restarts and uses its own kernel again.
Curious if you've made any progress on this? Or if there's a different kernel that will work on the SE so that I can finally get multicast routing working?
Multicast routing should already be supported in the stock UDM SE kernels.
Doesn't seem to be working for me and all I saw was a bunch of posts pointing to here about Multicast routing...is there something I need to enable that I'm missing? Multicast DNS seems to work, but not routing
For example here:
I have a new BGP multicast route between my two nodes of 10.45.0.12, but it times out any time I try to connect
@jimmy-ungerman You can verify it on your device as follows:
$ zgrep "MROUTE" /proc/config.gz
CONFIG_IP_MROUTE_COMMON=y
CONFIG_IP_MROUTE=y
# CONFIG_IP_MROUTE_MULTIPLE_TABLES is not set
# CONFIG_IPV6_MROUTE is not set
Make sure you actually have the multicast routes installed into the kernel.
@fabianishere I'm sorry I realized I've been saying "Multicast" when I meant "MultiPATH". Are you aware if this is enabled or not?
@jimmy-ungerman No, multipath routing is not supported by the stock kernel. You’ll need to use a custom kernel for that (if you search in the repository, you’ll find others also using it for that feature).