multus-cni
multus-cni copied to clipboard
Multus NIC cannot connect to the external NW.
What happend: I refer to https://github.com/Orange-OpenSource/towards5gs-helm , and I install helm. The free5GC helm install succeeds but cannot connect to the external NW. it appears that it is not tied to the Multus Master Interface. How to reslove this problems?
What you expected to happen: If the Multus Master Interface is connected to the Multus Master Interface, it is recognized that XXX@ifYYYis displayed, but it appears to be displayed incorrectly.
bash output
$ ip a
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 4c:52:62:65:bb:fd brd ff:ff:ff:ff:ff:ff
inet 172.31.16.37/24 brd 172.31.16.255 scope global eno1
valid_lft forever preferred_lft forever
inet6 fe80::4e52:62ff:fe65:bbfd/64 scope link
valid_lft forever preferred_lft forever
7: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
link/ether b8:ce:f6:72:b0:d4 brd ff:ff:ff:ff:ff:ff
inet 172.31.1.30/24 brd 172.31.1.255 scope global ens1f0
valid_lft forever preferred_lft forever
inet6 fe80::bace:f6ff:fe72:b0d4/64 scope link
valid_lft forever preferred_lft forever
$ helm install free5gc-v1 ./free5gc/ --set free5gc-upf.upf.securityContext.privileged=true --set free5gc-upf.upf.securityContext.capabilities=null --set global.n6network.subnetIP=172.31.1.0 --set global.n6network.gatewayIP=172.31.1.1 --set global.n6network.excludeIP=172.31.1.254 --set free5gc-upf.upf.n6if.ipAddress=172.31.1.11 --set global.n2network.masterIf=eno1 --set global.n3network.masterIf=eno1 --set global.n4network.masterIf=eno1 --set global.n6network.masterIf=ens1f0 --set global.n9network.masterIf=eno1
$ kubectl get po
NAME READY STATUS RESTARTS AGE
free5gc-v1-free5gc-amf-amf-69c476d9cb-k47kz 1/1 Running 0 52s
free5gc-v1-free5gc-ausf-ausf-7675996f67-ch2bt 1/1 Running 0 51s
free5gc-v1-free5gc-dbpython-dbpython-7d4c7fd7f8-zc254 1/1 Running 0 52s
free5gc-v1-free5gc-nrf-nrf-55cffd6c79-ft4w2 1/1 Running 0 51s
free5gc-v1-free5gc-nssf-nssf-795f9f5577-79rdz 1/1 Running 0 51s
free5gc-v1-free5gc-pcf-pcf-797f677cfc-kzwk4 1/1 Running 0 51s
free5gc-v1-free5gc-smf-smf-766cddcd99-skk95 1/1 Running 0 51s
free5gc-v1-free5gc-udm-udm-684f7b8688-p4drb 1/1 Running 0 52s
free5gc-v1-free5gc-udr-udr-65cdb74d7d-tv7ch 1/1 Running 0 50s
free5gc-v1-free5gc-upf-upf-57c6c59d8b-9qw8f 1/1 Running 0 51s
free5gc-v1-free5gc-webui-webui-5b9b94bb95-x2q4n 1/1 Running 0 52s
mongodb-0 1/1 Running 0 51s
$ kubectl exec -it free5gc-v1-free5gc-upf-upf-57c6c59d8b-9qw8f -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if51: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default
link/ether fe:2b:34:2a:32:a5 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 182.16.184.168/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::fc2b:34ff:fe2a:32a5/64 scope link
valid_lft forever preferred_lft forever
5: n3@tunl0: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether 4c:52:62:65:c5:d4 brd ff:ff:ff:ff:ff:ff
inet 10.100.50.233/29 brd 10.100.50.239 scope global n3
valid_lft forever preferred_lft forever
inet6 fe80::4c52:6200:165:c5d4/64 scope link
valid_lft forever preferred_lft forever
6: n6@n4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default
link/ether b8:ce:f6:7c:6b:58 brd ff:ff:ff:ff:ff:ff
inet 172.31.1.11/24 brd 172.31.1.255 scope global n6
valid_lft forever preferred_lft forever
inet6 fe80::b8ce:f600:17c:6b58/64 scope link
valid_lft forever preferred_lft forever
7: n4@tunl0: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether 4c:52:62:65:c5:d4 brd ff:ff:ff:ff:ff:ff
inet 10.100.50.241/29 brd 10.100.50.247 scope global n4
valid_lft forever preferred_lft forever
inet6 fe80::4c52:6200:265:c5d4/64 scope link
valid_lft forever preferred_lft forever
8: upfgtp: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1464 qdisc noqueue state UNKNOWN group default qlen 1000
link/none
inet6 fe80::c9a2:a8e0:6916:9641/64 scope link stable-privacy
valid_lft forever preferred_lft forever
Anything else we need to know?:
Environment:
- Multus version
v3.9.3
- Kubernetes version (use
kubectl version
):
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.16", GitCommit:"c5f43560a4f98f2af3743a59299fb79f07924373", GitTreeState:"clean", BuildDate:"2023-11-15T22:39:12Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.5", GitCommit:"804d6167111f6858541cef440ccc53887fbbc96a", GitTreeState:"clean", BuildDate:"2022-12-08T10:08:09Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
- Primary CNI for Kubernetes cluster:
cri-o
- OS (e.g. from /etc/os-release):
Ubuntu 20.04.1 LTS \n \l
- File of '/etc/cni/net.d/'
$ ls /etc/cni/net.d/
00-multus.conf 10-calico.conflist calico-kubeconfig multus.d
- File of '/etc/cni/multus/net.d'
$ ls /etc/cni/multus/net.d
ls: cannot access '/etc/cni/multus/net.d': No such file or directory
- NetworkAttachment info (use
kubectl get net-attach-def -o yaml
)
$ kubectl get net-attach-def n3network-free5gc-v1-free5gc-upf -o yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
annotations:
meta.helm.sh/release-name: free5gc-v1
meta.helm.sh/release-namespace: default
creationTimestamp: "2024-04-04T07:39:23Z"
generation: 1
labels:
app.kubernetes.io/managed-by: Helm
name: n3network-free5gc-v1-free5gc-upf
namespace: default
resourceVersion: "1353009"
uid: 5ec532e7-1348-4b2f-81d6-9f1f10221119
spec:
config: '{ "cniVersion": "0.3.1", "plugins": [ { "type": "ipvlan", "capabilities":
{ "ips": true }, "master": "eno1", "mode": "l2", "ipam": { "type": "static", "routes":
[ { "dst": "0.0.0.0/0", "gw": "10.100.50.238" } ] } } ] }'
$ kubectl get net-attach-def n4network-free5gc-v1-free5gc-upf -o yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
annotations:
meta.helm.sh/release-name: free5gc-v1
meta.helm.sh/release-namespace: default
creationTimestamp: "2024-04-04T07:39:23Z"
generation: 1
labels:
app.kubernetes.io/managed-by: Helm
name: n4network-free5gc-v1-free5gc-upf
namespace: default
resourceVersion: "1353008"
uid: e2d1e8b4-252f-431d-8777-b211bbd645e8
spec:
config: '{ "cniVersion": "0.3.1", "plugins": [ { "type": "ipvlan", "capabilities":
{ "ips": true }, "master": "eno1", "mode": "l2", "ipam": { "type": "static", "routes":
[ { "dst": "0.0.0.0/0", "gw": "10.100.50.246" } ] } } ] }'
$ kubectl get net-attach-def n6network-free5gc-v1-free5gc-upf -o yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
annotations:
meta.helm.sh/release-name: free5gc-v1
meta.helm.sh/release-namespace: default
creationTimestamp: "2024-04-04T07:39:23Z"
generation: 1
labels:
app.kubernetes.io/managed-by: Helm
name: n6network-free5gc-v1-free5gc-upf
namespace: default
resourceVersion: "1353007"
uid: 19e63d9b-6d5c-4544-8a2f-c3d3e021d31e
spec:
config: '{ "cniVersion": "0.3.1", "plugins": [ { "type": "ipvlan", "capabilities":
{ "ips": true }, "master": "ens1f0", "mode": "l2", "ipam": { "type": "static",
"routes": [ { "dst": "0.0.0.0/0", "gw": "172.31.1.1" } ] } } ] }'
- Target pod yaml info (with annotation, use
kubectl get pod <podname> -o yaml
)
apiVersion: v1 [167/1837]
kind: Pod
metadata:
annotations:
checksum/config: 9484d02a8722d94559244e9949507685e23a86b4b021a0cb65891810f2c8168e
cni.projectcalico.org/containerID: c92558def4e17e02399dba61ae607a0c4246642eb16429dd1a0c3512dd6fc872
cni.projectcalico.org/podIP: 182.16.182.141/32
cni.projectcalico.org/podIPs: 182.16.182.141/32
k8s.v1.cni.cncf.io/network-status: |-
[{
"name": "k8s-pod-network",
"ips": [
"182.16.182.141"
],
"default": true,
"dns": {}
},{
"name": "default/n3network-free5gc-v1-free5gc-upf",
"interface": "n3",
"ips": [
"10.100.50.233"
],
"mac": "4c:52:62:65:bb:fd",
"dns": {}
},{
"name": "default/n6network-free5gc-v1-free5gc-upf",
"interface": "n6",
"ips": [
"172.31.1.30"
],
"mac": "b8:ce:f6:72:b0:d4",
"dns": {}
},{
"name": "default/n4network-free5gc-v1-free5gc-upf",
"interface": "n4",
"ips": [
"10.100.50.241"
],
"mac": "4c:52:62:65:bb:fd",
"dns": {}
}]
k8s.v1.cni.cncf.io/networks: '[ { "name": "n3network-free5gc-v1-free5gc-upf", [126/1837]
"interface": "n3", "ips": [ "10.100.50.233/29" ], "gateway": [ "10.100.50.238"
] }, { "name": "n6network-free5gc-v1-free5gc-upf", "interface": "n6", "ips":
[ "172.31.1.30/24" ], "gateway": [ "172.31.1.1" ] }, { "name": "n4network-free5gc-v1-free5gc-upf",
"interface": "n4", "ips": [ "10.100.50.241/29" ], "gateway": [ "10.100.50.246"
] } ]'
k8s.v1.cni.cncf.io/networks-status: |-
[{
"name": "k8s-pod-network",
"ips": [
"182.16.182.141"
],
"default": true,
"dns": {}
},{
"name": "default/n3network-free5gc-v1-free5gc-upf",
"interface": "n3",
"ips": [
"10.100.50.233"
],
"mac": "4c:52:62:65:bb:fd",
"dns": {}
},{
"name": "default/n6network-free5gc-v1-free5gc-upf",
"interface": "n6",
"ips": [
"172.31.1.30"
],
"mac": "b8:ce:f6:72:b0:d4",
"dns": {}
},{
"name": "default/n4network-free5gc-v1-free5gc-upf",
"interface": "n4",
"ips": [
"10.100.50.241"
],
"mac": "4c:52:62:65:bb:fd",
"dns": {}
}]
creationTimestamp: "2024-04-04T07:39:23Z"
generateName: free5gc-v1-free5gc-upf-upf-857ff7ddcd-
labels:
app.kubernetes.io/instance: free5gc-v1
app.kubernetes.io/name: free5gc-upf
nf: upf
pod-template-hash: 857ff7ddcd
project: free5gc
name: free5gc-v1-free5gc-upf-upf-857ff7ddcd-gbdxw
namespace: default
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: free5gc-v1-free5gc-upf-upf-857ff7ddcd
uid: ee82588e-c8f2-409a-819c-727ae4138ebb
resourceVersion: "1353039"
uid: 1e267cc9-98ec-421b-a495-1d315c8ebe00
spec:
affinity: {}
containers:
- command:
- /free5gc/config//wrapper.sh
image: towards5gs/free5gc-upf:latest
imagePullPolicy: Always
name: upf
ports:
- containerPort: 8805
name: n4
protocol: UDP
resources:
requests:
cpu: 500m
memory: 512Mi
securityContext:
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts: [48/1837]
- mountPath: /free5gc/config/
name: free5gc-v1-free5gc-upf-upf-volume
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-mpnp5
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: machine-b1
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: free5gc-v1-free5gc-upf-upf-volume
projected:
defaultMode: 420
sources:
- configMap:
items:
- key: upfcfg.yaml
path: upfcfg.yaml
- key: wrapper.sh
mode: 511
path: wrapper.sh
name: free5gc-v1-free5gc-upf-upf-configmap
- name: kube-api-access-mpnp5
projected:
defaultMode: 420
sources:
- serviceAccountToken: [5/1837]
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2024-04-04T07:39:23Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2024-04-04T07:39:27Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2024-04-04T07:39:27Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2024-04-04T07:39:23Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: cri-o://d2de0e3ea96af3feec4345ca369f8f0a07d437f99e4b04ebd19ff763fc8942b8
image: docker.io/towards5gs/free5gc-upf:latest
imageID: docker.io/towards5gs/free5gc-upf@sha256:09ac580458cf960168d390224afa650bd0e6d9b3b534859e973b12f5708b7829
lastState: {}
name: upf
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2024-04-04T07:39:26Z"
hostIP: 172.31.16.37
phase: Running
podIP: 182.16.182.141
podIPs:
- ip: 182.16.182.141
qosClass: Burstable
startTime: "2024-04-04T07:39:23Z"
- Other log outputs (if you use multus logging)
This one seems to be rather implementation specific! If anyone wants to try to replicate using these components, help wanted for sure. Thanks!