Orange-OpenSource
Dev deployment on Minikube - SMF and UPF fail to listen on UDP addresses
Context
I am trying to deploy free5gc on local k8s cluster, so that I can explore it. I decided to spin a Minikube single-node cluster with Docker driver on my local Manjaro Linux laptop. It has following kernel: 5.15.60-1-MANJARO
Steps
Steps I performed:
-
Setup Minikube with Docker driver
-
Install gtp5g kernel module on your linux host
- Ensure linux-headers are installed
- Build and install the module:
it clone -b v0.6.5 https://github.com/free5gc/gtp5g.git d gtp5g ake udo make install -
Install multus-cni on your linux host
-
git clone https://github.com/k8snetworkplumbingwg/multus-cni.git && cd multus-cni -
cat ./deployments/multus-daemonset-thick-plugin.yml | kubectl apply -f -
-
-
Setup persistent volume in k8s
- Follow https://github.com/Orange-OpenSource/towards5gs-helm/blob/main/docs/demo/Setup-free5gc-and-test-with-UERANSIM.md#create-a-persistent-volume
-
minikube ssh->mkdir /home/docker/5gc-data - Create namespace 5gc and persistent volume
- Change PV node name in to minikube
- Change PV local path to /home/docker/5gc-data
-
Setup physical network interfaces in k8s node (eth0, eth1)
-
ip a show eth0on k8s node shows an interface - K8s node has no eth1 interface (should one be created?)
-
-
Install helm chart:
elm repo add towards5gs 'https://raw.githubusercontent.com/Orange-OpenSource/towards5gs-helm/main/repo/' helm repo update elm install 5gc-helm towards5gs/free5gc -n 5gc
K8s node has following eth0 interface:
docker@minikube:~$ ip a show eth0
10: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:c0:a8:31:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.49.2/24 brd 192.168.49.255 scope global eth0
valid_lft forever preferred_lft forever
```
K8s has no eth1 interface.
## Problem
SMF and UPF services fail:
```bash
NAME READY STATUS RESTARTS AGE
5gc-helm-free5gc-amf-amf-6949fcd8d-ktj4c 1/1 Running 0 5m25s
5gc-helm-free5gc-ausf-ausf-5cc7954594-qv4b4 1/1 Running 0 5m25s
5gc-helm-free5gc-nrf-nrf-6767465d47-2kl6d 1/1 Running 0 5m25s
5gc-helm-free5gc-nssf-nssf-7bcb45b6b9-v6xpf 1/1 Running 0 5m25s
5gc-helm-free5gc-pcf-pcf-56847599f4-xs6b6 1/1 Running 0 5m25s
5gc-helm-free5gc-smf-smf-79db7f6485-nb57b 0/1 CrashLoopBackOff 5 (110s ago) 5m25s
5gc-helm-free5gc-udm-udm-54fcb66c6b-86sl6 1/1 Running 0 5m25s
5gc-helm-free5gc-udr-udr-66b7d76f46-7w4wz 1/1 Running 0 5m25s
5gc-helm-free5gc-upf-upf-85c99f9dd9-snwp6 0/1 CrashLoopBackOff 5 (2m4s ago) 5m25s
5gc-helm-free5gc-webui-webui-bf5b9ff75-c5vfn 1/1 Running 0 5m25s
mongodb-0 1/1 Running 0 5m25s
➜ towards5gs-helm git:(main) ✗ kubectl -n 5gc logs pods/5gc-helm-free5gc-smf-smf-79db7f6485-nb57b
Defaulted container "smf" out of: smf, wait-nrf (init)
2022-09-08T14:14:43Z [INFO][SMF][CFG] SMF config version [1.0.2]
2022-09-08T14:14:43Z [INFO][SMF][CFG] UE-Routing config version [1.0.1]
2022-09-08T14:14:43Z [INFO][SMF][Init] SMF Log level is set to [info] level
2022-09-08T14:14:43Z [INFO][LIB][NAS] set log level : info
2022-09-08T14:14:43Z [INFO][LIB][NAS] set report call : false
2022-09-08T14:14:43Z [INFO][LIB][NGAP] set log level : info
2022-09-08T14:14:43Z [INFO][LIB][NGAP] set report call : false
2022-09-08T14:14:43Z [INFO][LIB][Aper] set log level : info
2022-09-08T14:14:43Z [INFO][LIB][Aper] set report call : false
2022-09-08T14:14:43Z [INFO][LIB][PFCP] set log level : info
2022-09-08T14:14:43Z [INFO][LIB][PFCP] set report call : false
2022-09-08T14:14:43Z [INFO][SMF][App] smf
2022-09-08T14:14:43Z [INFO][SMF][App] SMF version:
free5GC version: v3.2.0
build time: 2022-08-15T14:14:15Z
commit hash: de70bf6c
commit time: 2022-06-28T04:52:40Z
go version: go1.14.4 linux/amd64
2022-09-08T14:14:43Z [INFO][SMF][CTX] smfconfig Info: Version[1.0.2] Description[SMF initial local configuration]
2022-09-08T14:14:43Z [INFO][SMF][CTX] Endpoints: [10.100.50.233]
2022-09-08T14:14:43Z [INFO][SMF][Init] Server started
2022-09-08T14:14:43Z [INFO][SMF][Init] SMF Registration to NRF {1384474e-46bf-44a8-9c29-bfdae51157f3 SMF REGISTERED 0 0xc00028e240 0xc00028e2a0 [] [] [smf-nsmf] [] <nil> [] [] <nil> 0 0 0 area1 <nil> <nil> <nil> <nil> 0xc000131000 <nil> <nil> <nil> <nil> <nil> map[] <nil> false 0xc00028e060 false false []}
2022-09-08T14:14:43Z [ERRO][SMF][PFCP] Failed to listen: listen udp 10.100.50.244:8805: bind: cannot assign requested address
2022-09-08T14:14:43Z [FATA][SMF][App] panic: runtime error: invalid memory address or nil pointer dereference
goroutine 1 [running]:
runtime/debug.Stack(0xc0004b5628, 0xc7b0a0, 0x15067b0)
/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
main.main.func1()
/go/src/free5gc/NFs/smf/cmd/main.go:26 +0x57
panic(0xc7b0a0, 0x15067b0)
/usr/local/go/src/runtime/panic.go:969 +0x166
github.com/free5gc/smf/internal/pfcp/udp.Run(0xdd4700)
/go/src/free5gc/NFs/smf/internal/pfcp/udp/udp.go:27 +0x186
github.com/free5gc/smf/pkg/service.(*SMF).Start(0x1523eb0)
/go/src/free5gc/NFs/smf/pkg/service/init.go:267 +0x2b3
main.action(0xc000510580, 0x0, 0xc00002c240)
/go/src/free5gc/NFs/smf/cmd/main.go:65 +0x423
github.com/urfave/cli.HandleAction(0xc48000, 0xdd65d8, 0xc000510580, 0xc000510580, 0x0)
/go/pkg/mod/github.com/urfave/[email protected]/app.go:524 +0x11a
github.com/urfave/cli.(*App).Run(0xc0005241c0, 0xc0000be000, 0x5, 0x5, 0x0, 0x0)
/go/pkg/mod/github.com/urfave/[email protected]/app.go:286 +0x649
main.main()
/go/src/free5gc/NFs/smf/cmd/main.go:37 +0x188
➜ towards5gs-helm git:(main) ✗ kubectl -n 5gc logs pods/5gc-helm-free5gc-upf-upf-85c99f9dd9-snwp6
Cannot find device "n6"
2022-09-08T14:19:29Z [INFO][UPF][Main] UPF version:
free5GC version: v3.2.0
build time: 2022-08-15T14:14:32Z
commit hash: 4972fffb
commit time: 2022-06-29T05:46:33Z
go version: go1.14.4 linux/amd64
2022-09-08T14:19:29Z [INFO][UPF][Cfg] Read config from [/free5gc/config//upfcfg.yaml]
2022-09-08T14:19:29Z [INFO][UPF][Cfg] ==================================================
2022-09-08T14:19:29Z [INFO][UPF][Cfg] (*factory.Config)(0xc0000d4000)({
Version: (string) (len=5) "1.0.3",
Description: (string) (len=31) "UPF initial local configuration",
Pfcp: (*factory.Pfcp)(0xc00009e8d0)({
Addr: (string) (len=13) "10.100.50.241",
NodeID: (string) (len=13) "10.100.50.241",
RetransTimeout: (time.Duration) 1s,
MaxRetrans: (uint8) 3
}),
Gtpu: (*factory.Gtpu)(0xc00009ea80)({
Forwarder: (string) (len=5) "gtp5g",
IfList: ([]factory.IfInfo) (len=1 cap=1) {
(factory.IfInfo) {
Addr: (string) (len=13) "10.100.50.233",
Type: (string) (len=2) "N3",
Name: (string) "",
IfName: (string) ""
}
}
}),
DnnList: ([]factory.DnnList) (len=1 cap=1) {
(factory.DnnList) {
Dnn: (string) (len=8) "internet",
Cidr: (string) (len=11) "10.1.0.0/17",
NatIfName: (string) (len=2) "n6"
}
},
Logger: (*factory.Logger)(0xc0000a4680)({
Enable: (bool) true,
Level: (string) (len=4) "info",
ReportCaller: (bool) false
})
})
2022-09-08T14:19:29Z [INFO][UPF][Cfg] ==================================================
2022-09-08T14:19:29Z [INFO][UPF][Main] Log level is set to [info] level
2022-09-08T14:19:29Z [INFO][UPF][Main] starting Gtpu Forwarder [gtp5g]
2022-09-08T14:19:29Z [INFO][UPF][Main] GTP Address: "10.100.50.233:2152"
2022-09-08T14:19:29Z [ERRO][UPF][Main] UPF Cli Run Error: open Gtp5g: open link: listen: listen udp 10.100.50.233:2152: bind: cannot assign requested address
Adjusting several helm variables didn't bring an improvement:
helm upgrade 5gc-helm towards5gs/free5gc -n 5gc --set global.n6network.masterIf=eth0 --set global.n6network.subnetIP=192.168.49.0 --set global.n6network.gatewayIP=192.168.49.1 --set free5gc-upf.upf.n6if.ipAddress=192.168.49.2
Questions
- Does this dev deployment make generally sense? E.g. can free5gc work on newer kernels?
- Did I miss some setup steps?
- What adjustments should I make here? Should I create eth1 interface or change more helm values, e.g.
upf.n3if.ipAddress?
Thank you in advance.
@RaulB16 No, I didn't proceed further with this setup. I ended up using AWS EKS with Ubuntu 20.04 node and it worked much better. I needed to only change network interfaces names (e.g. n2network.masterIf) in free5gc Helm values file (to ens5 in my case).
Hi @danielfurman and @RaulB16
Honestly, I don't know where to install the gtp kernel module when using Minikube. I think you have to do it at the host level since you are using the Docker driver to create your cluster, but it doesn't seem to work. I leave this issue open to see if there is already someone who managed to deploy the project on Minikube.
Can you still share the state of the UPF and SMF pods with the kubectl describe command please? just to check that the N3 and N4 interfaces are created.
Raouf
Hi, with a help of my colleague I managed to proceed further with Minikube deployment.
I don't have the first deployment on my machine, but the problem was that Macvlan interfaces (e.g. N3, N4) were not created due to problem with CNIs configuration. I'll show better approach below.
Setup 2 - steps
-
Start Minikube cluster with Docker driver
# Starting minikube with Multus CNI minikube start --memory 4096 --cni=/home/daniel/projects/vmw/multus-cni/deployments/multus-daemonset.yml # Fix bridge CNI IPv6 issue (see: https://github.com/cri-o/cri-o/issues/3555#issuecomment-1027615192) minikube ssh sysctl -w net.ipv6.conf.all.disable_ipv6=0 sysctl -w net.ipv6.conf.default.disable_ipv6=0 -
Install gtp5g kernel module on your linux host (same as before)
-
Create 5gc and ueran namespaces
kubectl create namespace 5gc kubectl create namespace ueran -
Optional - deploy Istio
-
Deploy free5gc with Helm
helm install free5gc towards5gs/free5gc -n 5gc -f ./free5gc-k8s-cluster/free5gc-helm-mk.yamlglobal: n2network: masterIf: eth0 # eth0 is the interface you can find on cluster Node n3network: masterIf: eth0 n4network: masterIf: eth0 n6network: masterIf: eth0 n9network: masterIf: eth0 -
Register UE via free5gc webui
- Get k8s node external IP via following command:
export NODE_IP=$(minikube ip) - Navigate to free5gc webui:
firefox ${NODE_IP}:30500(user: admin, pass: free5gc) - Add a new subscriber (UE) with form's default values: Subscribers -> New Subscriber -> Submit
- Get k8s node external IP via following command:
-
Deploy UERANSIM with Helm
helm install ueransim towards5gs/ueransim -n ueran -f ./free5gc-k8s-cluster/ueransim-helm-mk.yamlglobal: n2network: masterIf: eth0 n3network: masterIf: eth0
Setup 2 - status
-
Free5gc pods are running but some (e.g. NSSF) are logging several NRF connection errors: pod-errors.log
-
Macvlan interfaces are created in pods
-
UE established PDU session and created uesimtun0 interface
-
UPF pod has ip forwarding enabled:
➜ 5g git:(master) ✗ kubectl -n 5gc exec -it pod/free5gc-free5gc-upf-upf-566cdc4f54-b7zn7 -- cat /proc/sys/net/ipv4/ip_forward 1 -
UE ping does not work due to
Destination Host Unreachable➜ 5g git:(master) ✗ kubectl -n ueran exec -it $UE_POD_NAME -- ping -I uesimtun0 www.google.com PING www.google.com (142.250.186.36) from 10.1.0.1 uesimtun0: 56(84) bytes of data. From 10.100.100.12 (10.100.100.12) icmp_seq=1 Destination Host Unreachable From 10.100.100.12 (10.100.100.12) icmp_seq=2 Destination Host Unreachable -
More info: kubectl-get-all.txt upf-pod.txt ue.log ue-pod.txt minikube.md - ip address and route info on pods, node, and laptop
Summary
If anyone has an idea why UE-internet ping might not be working or how to debug that, please let me know.
I managed to fix UE-internet ping for the Minikube setup. The last problem was that the following condition was not fullfiled (https://github.com/Orange-OpenSource/towards5gs-helm/tree/main/charts/free5gc#networks-configuration):
In addition, please make sure global.n6network.subnetIP, global.n6network.gatewayIP and free5gc-upf.upf.n6if.ipAddress parameters will match the IP address of the toto interface in order to make the UPF able to reach the Data Network via its N6 interface.
To fulfill that condition the easiest way was to start Minikube with a subnet matching N6 network default parameters:
minikube start --memory 8192 --subnet=10.100.100.0/24 --cni=../multus-cni/deployments/multus-daemonset.yml
The rest of steps are the same as in Setup 2
Since the connectivity test is working for me, I am closing the issue.