linux
linux copied to clipboard
raspberrypi
No loadable module for xfrm_user / xfrm_interface
Describe the bug
There is no loadable module for xfrm_user which is needed for proper Cilium support.
Steps to reproduce the behaviour
- Try to use Cillium in k8s.
- Have it crash and start up properly.
Device (s)
Raspberry Pi 4 Mod. B
System
OS: Ubuntu Server 22.04 LTS
Firmware:
Nov 18 2021 16:16:49
Copyright (c) 2012 Broadcom
version d9b293558b4cef6aabedcc53c178e7604de90788 (clean) (release) (sta
Kernel version: Linux k8s-worker7 5.15.0-1013-raspi #15-Ubuntu SMP PREEMPT Mon Aug 8 06:33:06 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
Logs
No response
Additional context
Related: https://github.com/cilium/cilium/issues/20901
xfrm_user doesn't exist as a module because it is built-in (surprisingly). However, CONFIG_XFRM_INTERFACE is not enabled.
Please obtain a complete list of settings that are required on Raspberry Pis and we'll consider adding them.
Full list:
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_NET_CLS_BPF=y
CONFIG_BPF_JIT=y
CONFIG_NET_CLS_ACT=y
CONFIG_NET_SCH_INGRESS=y
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_USER_API_HASH=y
CONFIG_CGROUPS=y
CONFIG_CGROUP_BPF=y
CONFIG_NETFILTER_XT_SET=m
CONFIG_IP_SET=m
CONFIG_IP_SET_HASH_IP=m
CONFIG_NETFILTER_XT_TARGET_TPROXY=m
CONFIG_NETFILTER_XT_TARGET_CT=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_SOCKET=m
CONFIG_XFRM=y
CONFIG_XFRM_OFFLOAD=y
CONFIG_XFRM_STATISTICS=y
CONFIG_XFRM_ALGO=m
CONFIG_XFRM_USER=m
CONFIG_INET{,6}_ESP=m
CONFIG_INET{,6}_IPCOMP=m
CONFIG_INET{,6}_XFRM_TUNNEL=m
CONFIG_INET{,6}_TUNNEL=m
CONFIG_INET_XFRM_MODE_TUNNEL=m
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=m
CONFIG_CRYPTO_GCM=m
CONFIG_CRYPTO_SEQIV=m
CONFIG_CRYPTO_CBC=m
CONFIG_CRYPTO_HMAC=m
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_AES=m
CONFIG_NET_SCH_FQ=m
@pelwell sorry new to raspberrypi, does raspberry pi kernel enable kernel config support for /proc/config.gz
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
does raspberry pi kernel enable kernel config support for /proc/config.gz
Yes, but you must "sudo modprobe configs" first.
That looks like a reasonable list of config settings. Enabling BPF_JIT is the largest cost, and that adds less than 18kB to the static kernel, and it that is slightly mitigated by setting XFRM_USER=m instead.
Is it reasonable to restrict this to BCM2711/Pi 4 defconfigs? And what about 64-bit vs 32-bit?
For the record, this (slightly) tweaked list of settings:
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_NET_CLS_BPF=y
CONFIG_BPF_JIT=y
CONFIG_NET_CLS_ACT=y
CONFIG_NET_SCH_INGRESS=m
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_USER_API_HASH=m
CONFIG_CGROUPS=y
CONFIG_CGROUP_BPF=y
CONFIG_NETFILTER_XT_SET=m
CONFIG_IP_SET=m
CONFIG_IP_SET_HASH_IP=m
CONFIG_NETFILTER_XT_TARGET_TPROXY=m
CONFIG_NETFILTER_XT_TARGET_CT=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_SOCKET=m
CONFIG_XFRM=y
CONFIG_XFRM_OFFLOAD=y
CONFIG_XFRM_STATISTICS=y
CONFIG_XFRM_ALGO=m
CONFIG_XFRM_USER=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
CONFIG_INET6_ESP=m
CONFIG_INET6_IPCOMP=m
CONFIG_INET6_XFRM_TUNNEL=m
CONFIG_INET6_TUNNEL=m
# CONFIG_INET_XFRM_MODE_TUNNEL=m Does not exist
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_GCM=m
CONFIG_CRYPTO_SEQIV=m
CONFIG_CRYPTO_CBC=m
CONFIG_CRYPTO_HMAC=m
CONFIG_CRYPTO_SHA256=y
CONFIG_CRYPTO_AES=m
CONFIG_NET_SCH_FQ=m
CONFIG_IKCONFIG=m
CONFIG_IKCONFIG_PROC=y
results in this much smaller list of defconfig changes:
index 2c8e5e0cdab6..17246b634d6d 100644
--- a/arch/arm64/configs/bcm2711_defconfig
+++ b/arch/arm64/configs/bcm2711_defconfig
@@ -6,6 +6,7 @@ CONFIG_GENERIC_IRQ_DEBUGFS=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_BPF_SYSCALL=y
+CONFIG_BPF_JIT=y
CONFIG_PREEMPT=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
@@ -77,7 +78,8 @@ CONFIG_ZSMALLOC=m
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_UNIX=y
-CONFIG_XFRM_USER=y
+CONFIG_XFRM_USER=m
+CONFIG_XFRM_STATISTICS=y
CONFIG_NET_KEY=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
@@ -379,6 +381,7 @@ CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
CONFIG_NET_CLS_FLOW=m
CONFIG_NET_CLS_CGROUP=m
+CONFIG_NET_CLS_BPF=y
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_CMP=m
CONFIG_NET_EMATCH_NBYTE=m
@@ -1564,9 +1567,12 @@ CONFIG_SECURITY_APPARMOR=y
CONFIG_LSM=""
CONFIG_CRYPTO_USER=m
CONFIG_CRYPTO_CHACHA20POLY1305=m
+CONFIG_CRYPTO_CBC=m
CONFIG_CRYPTO_ADIANTUM=m
+CONFIG_CRYPTO_HMAC=m
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_WP512=m
+CONFIG_CRYPTO_AES=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_DES=y
CONFIG_CRYPTO_LZ4=m
Is it reasonable to restrict this to BCM2711/Pi 4 defconfigs? And what about 64-bit vs 32-bit?
Yes, that's completely reasonable and 64bit only is a good target.
See 91b21aa.
Are you able to compile a kernel to verify that it works with this patch? You will find instructions here: https://www.raspberrypi.com/documentation/computers/linux_kernel.html#building
I have built 5.15.61-v8 (1.20220830) and found the following conflicts:
- Memory cgroups are not enabled, I had to modify the cmdline.txt adding
group_memory=1 cgroup_enable=memory - Cilium still fails to start but this time is an envoy proxy error:
level=fatal msg="Envoy: Binary \"cilium-envoy\" cannot be executed" error="signal: aborted (core dumped)" subsys=envoy-manager
Investigating a bit, I found a couple of relevant issues:
- https://github.com/cilium/cilium/issues/17467
- https://github.com/envoyproxy/envoy/issues/15235
Apart from that, I added CONFIG_BLK_DEV_RBD=m when compiling the kernel for rook.io to work, but that's a different topic.
Is there anything we can add to the kernel config for 2711 to fix the envoy issue? Thank you for your time!
EDIT: found a similar issue #4375
