danm
danm copied to clipboard
Published
20 hours ago •
nokia
nokia
IP Address allocation fails
Is this a BUG REPORT or FEATURE REQUEST?:
Uncomment only one, leave it on its own line:
bug
feature
What happened: Deployed a SRIOV-Network with 3 ips and attached ip's statically from that network to every pod. Sometimes I have observed an issue where one of the pod doesn't come up. (error is : ip address is already in use)
What you expected to happen: All the three pods should come up
How to reproduce it: Deploy the sriov network with 3 ips. Deploy three pods. one of the pod doesn't come up. It says that "IP address is already in use"
Anything else we need to know?:
kubectl describe cn sriov-o1c
.....
.....
allocation_pool:
End: 10.208.122.88
Last Ip:
Start: 10.208.122.86
controller-1:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
**florida-4005cucp-59958f6694-hngjj** 7/7 Running 0 21h
**florida-4005cuup1-65575b9c45-6bl7s** 5/5 Running 0 21h
**florida-4005du1-56f964cd49-cndj2** 0/8 ContainerCreating 0 21h
controller-1:~# kubectl exec -it **florida-4005cucp-59958f6694-hngjj** sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
Defaulting container name to gnb-cu-cp-app.
Use 'kubectl describe pod/florida-4005cucp-59958f6694-hngjj -n default' to see all of the containers in this pod.
sh-4.2# ip a s neto1c
39: neto1c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 1a:57:2c:00:80:64 brd ff:ff:ff:ff:ff:ff
inet 10.208.122.86/24 brd 10.208.122.255 scope global neto1c
valid_lft forever preferred_lft forever
sh-4.2# exit
controller-1:~# kubectl exec -it **florida-4005cuup1-65575b9c45-6bl7s** sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
Defaulting container name to gnb-cu-up-pdcp.
Use 'kubectl describe pod/florida-4005cuup1-65575b9c45-6bl7s -n default' to see all of the containers in this pod.
sh-4.2# ip a s neto1c
16: neto1c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether ca:9b:d5:67:47:22 brd ff:ff:ff:ff:ff:ff
inet 10.208.122.87/24 brd 10.208.122.255 scope global neto1c
valid_lft forever preferred_lft forever
sh-4.2# exit
controller-1:~# kubectl describe pod florida-4005
Warning FailedCreatePodSandBox 21h kubelet, controller-1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fdbbfa43a2da679d08734c9b5860e25085a5c8c22054e0779ff052ca8989f972": CNI network could not be set up: CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:10.208.122.88 is already in use
I have not deployed any pods in any other namespace which is using this ip "10.208.122.88". This issue is seen sometimes.
Environment:
-
DANM version (use
danm -version): controller-1:~# /usr/libexec/cni/danm --version 2021/05/20 07:22:22 DANM binary was built from release: v4.2.1 2021/05/20 07:22:22 DANM binary was built from commit: abd3c48d_dirty -
Kubernetes version (use
kubectl version): controller-1:~# kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"archive", BuildDate:"2020-08-05T05:08:32Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-08T17:30:47Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} -
DANM configuration (K8s manifests, kubeconfig files, CNI config file):
controller-1:~# cat /etc/cni/net.d/00-danm.conf
{
"cniVersion": "0.3.1",
"name": "danm_meta_cni",
"type": "danm",
"kubeconfig": "/etc/cni/net.d/danm-kubeconfig",
"cniDir": "/etc/cni/net.d",
"namingScheme": "legacy"
}
controller-1:~# cat /etc/cni/net.d/danm-kubeconfig
---
apiVersion: v1
kind: Config
current-context: default
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeE1EUXhPVEV3TVRreU5sb1hEVE14TURReE56RXdNVGt5Tmxvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTUdhCmlEcnhUWGxhQmJ0NWJpc1F6REdleVhkK1dhdWZBWkQrSUtIUm05bER4eUVaMExmZS9PMkxEUHRGM0NhNmZGV1cKajFZYjNNVVlLNlMrQ2loZGJnSmFYT2pFNllSalhPVFplZ0FRSlpoRUhROURQdGhQM2d1cUVtN2xsYWI3THVURgpFbUU0bU1pc1hOZGp6Q0lPTkVOeGVFeXl5dFFTODhKRnhBNFRpb2NFTjZnOFlsQXV4eDc2V2lCbHNSV3N2T1EyCjhXQlY2UnE4RFNmNWw1QTR2UTNmRm9lU0pIaHpsaDNXTlkzN1l0ekQvVlkzRUQyRjNpb1owYjZNRVFPdzZWSWUKMXl5b1JnbUtNUFo0ZFVIWUdnN1ZDQVZ2alJLR29QSlRSMzZ2cE5sVGVUZVE4UFhTQnYrNlZBUUZET3d4UWNoSgpiWUgzZkNzbFp2SW0zSUZQN3JjQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFLU2J5aXF1ZjJ6ZU14S0Urd1dGY0M0aEp5dUUKWGNXV2g4VExvanpFamdxbUk5VUZpbU9PNllyRFdHR01sblQ0OTJhM3RYT2llR2crZnltVVpOelZuanc3Vk13aQppc21tS25lVFBjSEQrTzFOZmJwUlRwWUdtOFhTOGJBNG1HbGJlTVNsMTdiK1hnRkZFVDRDOWo4U1hlMVJ4bnFiCi9wbmx6Y3h4N0xoaGRPcmFPZXByQ0o3aUFZOWZuSVE0Ym50OS9LUE93ZUpCaW1HeHdPWFN2N05KWCtBaWxPTWwKWURBMGZQclhUcGhWd2Qwb1pKd1c0TXVLdkwrd3NpMnFyWU8yVzQ4NXNOVklNTER6K0NEOEhBL2RxblhUdGpKTQpVaGNDcHJ0Y2lJOHh1VmYvb0hGUDFycE1GSVlKMlY1eStoaGtESDk2Tmt2MGpUM0NwZlFwOEFaVFBNND0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==
server: https://10.96.0.1:443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: danm
name: default
users:
- name: danm
user:
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IlFTcUlPdjVCREZrQkdLLVBmSjB5TDYyUEFyZFR5TktvMkNIb1Z6SEVtU2MifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYW5tLXRva2VuLWNjNWxtIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImRhbm0iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIwN2ZjZDI3YS04MjYxLTQ4OGQtODdlZi01NzQxNDYxZTlhY2QiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06ZGFubSJ9.TbMHYyYnifdLv57HuHU0H_4tsrBsdqFcgzIKtui6o9wz7W9AY7Eg0dyKCmNh2YAziu_izI-1m0zVN-Lgy46jbUiYBGlZygvyfE3yMhBtsm3rGbHk1Cz7vUcSk6D5FToQfJueyJtNAF53_qfsccTe_iQgZzBTMRpNFFCm8lRolC-kwt9vlJ4CX4faEofP4GSMUh89VzmMsByHGBnRgvJUnZ5tSxyZyitnISiR6U3r5MAuxJMy0qL8Gbcu1J9VePSOmcxmgUB8UhHfZXe59WyPQccN_F0p5-4adMqlApy7fwoCOq0m-Mn1MAuZdWwmeyUUnkHR9JWaTMR91B_xadqjHQ
preferences: {}
- OS (e.g. from /etc/os-release):
controller-1:~# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
-
Kernel (e.g.
uname -a): controller-1:~# uname -a Linux controller-1 4.18.0-147.3.1.rt24.96.el8_1.tis.8.x86_64 #1 SMP PREEMPT RT Wed Aug 5 06:21:07 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux -
Others:
ye we recently observed the same, but with IPVLAN. I have no idea how is this possible after first glance, because this error is thrown by the kernel itself, not by DANM which means the address is physically really in use on the server, in one of the namespaces, there is no other explanation for it
sry your issue might be a little bit different, as it indeed comes from IPAM not the kernel. our error is thrown when we try to allocate an already reserved IP to an IPVLAN slave, and kernel says it cannot do that the root cause might be the same. will def investigate!
so, our issue is different and I solved it in the other PR
your issue I can't reproduce at all in my environment. I deploy a network with an allocation pool of 3 IPs, I create 3 deployments with one Pod each from the same file, pin all three on the same node, I ask a different static IP for all three Pods: I have 100% success rate
please share the output of kubectl get danmeps command when the issue happens
This issue is observed once in a while. I will take the o/p of the command you asked, when the issue is reproduced. Thanks for looking at it.
@sriramec any news? reason why the DanmEp listing would be interesting cause I suspect we might be dealing with a "normal" case of synching issues. DANM Cleaner was created to address such issues, and I strongly recommend using it in production. let's see if it would help on your case, or this is something else
controller-0:~# kubectl get danmeps
NAME AGE
003789c0-171c-484f-904e-e53c9f4fc144 4d8h
045584d3-a8b1-48c7-9f46-a8428a7df892 4d8h
067ebf43-5ae0-4c1e-b7d8-59d2946f9d81 4d8h
0df739da-4271-457c-b38f-374e312af3e0 4d8h
144646af-5e81-4d86-835a-1fd3d44bcb79 4d8h
1594fa77-fa34-4dc4-896c-5398da3990b4 4d8h
24629bf3-619d-4494-8a6f-79d0c6fba26d 4d8h
30031531-785d-43fd-a43b-a39cbbc193f9 4d8h
3f0c3b15-74a3-4179-a898-bffa267fb8d3 149m
493ef085-9741-4119-8d0d-238cce1ea61b 2s
563bd99f-681f-474a-bee0-d322a8c731dd 4d8h
5879a061-0027-401b-98e1-aafc9e9d5074 25h
6c6097ad-4990-4324-8e82-d85f9518fcaa 4d8h
7e85a1d5-d248-40a9-9cf2-8058a9124ab5 4d8h
7f62bd64-c899-4662-8cb6-12f3035cc6de 4d8h
9aa1433e-3a49-4501-9b34-2f91574cc690 2s
b2a1ad70-34ba-4e4e-9439-3cc9280c26aa 2s
b810cce5-fc55-4a53-9cc5-2732f06071f1 4d8h
c6ef4f33-4a34-4d6e-a95a-07f658fac9ec 25h
d5f2f02d-c171-4edd-98c1-be94f7b54b06 4d8h
f984cd85-8cc2-4719-b0e9-42e5995ba44d 4d8h
controller-0:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
gnb11svt-1100cucp1-7dd8d99884-wzdz9 0/7 ContainerCreating 0 128m
controller-0:~# kubectl describe pod gnb11svt-1100cucp1-7dd8d99884-wzdz9
Annotations: cni.projectcalico.org/podIP: 172.16.192.133/32
cni.projectcalico.org/podIPs: 172.16.192.133/32
danm.k8s.io/interfaces:
[
{ "clusterNetwork": "default", "ip": "dynamic" },
{ "clusterNetwork": "sriov-x2c", "ip6": "2001:11:195:101::101", "proutes6": {"2001:11:195:137::/64": "2001:11:195:101::1"} },
{ "clusterNetwork": "sriov-f1c", "ip6": "dynamic" },
{ "clusterNetwork": "sriov-e1c", "ip6": "dynamic" },
{ "clusterNetwork": "sriov-o1c", "ip6": "2001:11:194:101::101", "proutes6": {"2001:31:212:2::/64": "2001:11:194:101::1", "2001:11:194:10...
]
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/gnb11svt-1100cucp1-7dd8d99884
Containers:
gnb-cu-cp-app:
Container ID:
....
....
Image ID:
Port: <none>
Host Port: <none>
....
....
Limits:
cpu: 250m
intel.com/pci_sriov_net_physnet0: 4
memory: 2G
Requests:
cpu: 250m
intel.com/pci_sriov_net_physnet0: 4
memory: 2G
default-token-b4cnt:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-b4cnt
Optional: false
QoS Class: Guaranteed
Node-Selectors: gnbnodename=cumachine
kubernetes.io/hostname=cumachine-1100
Tolerations: node.kubernetes.io/not-ready:NoExecute for 30s
node.kubernetes.io/unreachable:NoExecute for 30s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 128m (x3 over 128m) default-scheduler 0/2 nodes are available: 2 Insufficient intel.com/pci_sriov_net_physnet0.
Warning FailedScheduling 108m (x17 over 128m) default-scheduler 0/2 nodes are available: 2 node(s) didn't match node selector.
Normal Scheduled 108m default-scheduler Successfully assigned default/gnb11svt-1100cucp1-7dd8d99884-wzdz9 to controller-0
Warning FailedCreatePodSandBox 108m kubelet, controller-0 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8dcd995eedaa3f028876f8d41ecccdbc97cdc5ba5904ca93419619c48a99e3fa": CNI network could not be set up: CNI operation for network:sriov-x2c failed with:IP address reservation failed for network:sriov-x2c with error:failed to allocate IP address for network:sriov-x2c with error:static IP allocation failed, requested IP address:2001:11:195:101::101 is already in use
CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::101 is already in use
Warning FailedCreatePodSandBox 108m kubelet, controller-0 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0ad9315f4ddaf62c5b1c933c50e73d33976c564faa9a572573b945a7f8800c92": CNI network could not be set up: CNI operation for network:sriov-x2c failed with:IP address reservation failed for network:sriov-x2c with error:failed to allocate IP address for network:sriov-x2c with error:static IP allocation failed, requested IP address:2001:11:195:101::101 is already in use
CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::101 is already in use
Warning FailedCreatePodSandBox 107m kubelet, controller-0 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "263f9bce05623ecfdea4f42a7edf95828cd76af255df6e8616c382948eec193c": CNI network could not be set up: CNI operation for network:sriov-x2c failed with:IP address reservation failed for network:sriov-x2c with error:failed to allocate IP address for network:sriov-x2c with error:static IP allocation failed, requested IP address:2001:11:195:101::101 is already in use
CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::101 is already in use
Warning FailedCreatePodSandBox 107m kubelet, controller-0 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ae9754e5e86713bbd188cf9c0073848472d0e12ea29148c1e611dd37861152b2": CNI network could not be set up: CNI operation for network:sriov-x2c failed with:IP address reservation failed for network:sriov-x2c with error:failed to allocate IP address for network:sriov-x2c with error:static IP allocation failed, requested IP address:2001:11:195:101::101 is already in use
CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::101 is already in use
Warning FailedCreatePodSandBox 107m kubelet, controller-0 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9995646806d2dfd1b09868962e4cad1f894c16f58dac36893c852d94c5358af3": CNI network could not be set up: CNI operation for network:sriov-x2c failed with:IP address reservation failed for network:sriov-x2c with error:failed to allocate IP address for network:sriov-x2c with error:static IP allocation failed, requested IP address:2001:11:195:101::101 is already in use
CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::101 is already in use
Warning FailedCreatePodSandBox 107m kubelet, controller-0 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2e3049d588fe497e6008e96fe67284f4dc9e606e3211aaab8ddfdf8ae8538aff": CNI network could not be set up: CNI operation for network:sriov-x2c failed with:IP address reservation failed for network:sriov-x2c with error:failed to allocate IP address for network:sriov-x2c with error:static IP allocation failed, requested IP address:2001:11:195:101::101 is already in use
CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::101 is already in use
Warning FailedCreatePodSandBox 107m kubelet, controller-0 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "54b30352763d9a813f068f26788f8c57c3c36b7003e21e5a84e519e5228ac767": CNI network could not be set up: CNI operation for network:sriov-x2c failed with:IP address reservation failed for network:sriov-x2c with error:failed to allocate IP address for network:sriov-x2c with error:static IP allocation failed, requested IP address:2001:11:195:101::101 is already in use
CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::101 is already in use
Warning FailedCreatePodSandBox 107m kubelet, controller-0 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "48979a579e51789e0e97fe7b61307b198f5011fd9a1eef71dfa8493779002f63": CNI network could not be set up: CNI operation for network:sriov-x2c failed with:IP address reservation failed for network:sriov-x2c with error:failed to allocate IP address for network:sriov-x2c with error:static IP allocation failed, requested IP address:2001:11:195:101::101 is already in use
CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::101 is already in use
Warning FailedCreatePodSandBox 107m kubelet, controller-0 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "21581fcb37c331bde8ebe45d42468d66638fabf54a3d9b8cc6fe69993d8eb1e6": CNI network could not be set up: CNI operation for network:sriov-x2c failed with:IP address reservation failed for network:sriov-x2c with error:failed to allocate IP address for network:sriov-x2c with error:static IP allocation failed, requested IP address:2001:11:195:101::101 is already in use
CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::101 is already in use
Warning FailedCreatePodSandBox 3m5s (x2013 over 107m) kubelet, controller-0 (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "461a1a20ba48488f897029b8e5c520cea44dbf4a7c79b09309253da763fca5bf": CNI network could not be set up: CNI operation for network:sriov-x2c failed with:IP address reservation failed for network:sriov-x2c with error:failed to allocate IP address for network:sriov-x2c with error:static IP allocation failed, requested IP address:2001:11:195:101::101 is already in use
CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::101 is already in use
ye so basically confirmed what I explained above. the TL;DR version is that you should install DANM Cleaner in your environment, and you won't see this issue
I installed the danm cleaner in the setup, where the issue is seen and redeployed the pods. but still seeing the same issue Installed this cleaner.yaml and issue logs are seen below. Any suggestions to be overcome this problem. Is this cleaner.yaml file good enough.
controller-0:~/sriram$ cat cleaner.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: danm-cleaner
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: caas:danm-cleaner
rules:
- apiGroups:
- ""
resources:
- pods
verbs:
- list
- watch
- get
- apiGroups:
- ""
resources:
- events
- endpoints
verbs:
- create
- update
- get
- apiGroups:
- danm.k8s.io
resources:
- danmnets
- danmeps
- tenantnetworks
- clusternetworks
verbs: [ "*" ]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: caas:danm-cleaner
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: caas:danm-cleaner
subjects:
- kind: ServiceAccount
name: danm-cleaner
namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: danm-cleaner
namespace: kube-system
labels:
danm: cleaner
spec:
selector:
matchLabels:
danm: cleaner
template:
metadata:
name: danm-cleaner
labels:
danm: cleaner
spec:
serviceAccountName: danm-cleaner
containers:
- name: danm-cleaner
image: registry.local:9001/cleaner:latest
imagePullPolicy: IfNotPresent
controller-0:~/sriram$ kubectl get pods -n kube-system -o wide | grep -i cleaner
danm-cleaner-4gbzf 1/1 Running 0 113s 172.16.166.185 controller-1 <none> <none>
danm-cleaner-s7zxz 1/1 Running 0 113s 172.16.192.95 controller-0 <none> <none>
controller-0:~/sriram$ kubectl describe pod gnb12svt-1200du1-546d865594-456l5
Name: gnb12svt-1200du1-546d865594-456l5
Namespace: default
Priority: 0
Node: controller-1/10.215.12.3
Start Time: Wed, 30 Jun 2021 08:00:37 +0000
Labels: app.kubernetes.io/instance=gnb12svt-1200du1
app.kubernetes.io/name=cs-gnb-du
pod-template-hash=546d865594
Annotations: cni.projectcalico.org/podIP: 172.16.166.153/32
cni.projectcalico.org/podIPs: 172.16.166.153/32
danm.k8s.io/interfaces:
[
{ "clusterNetwork": "default", "ip": "dynamic" },
{ "clusterNetwork": "sriov-dpdk", "ip6": "dynamic" },
{ "clusterNetwork": "sriov-f1c", "ip6": "dynamic" },
{ "clusterNetwork": "sriov-o1c", "ip6": "2001:11:194:101::113", "proutes6": {"2001:31:212:2::/64": "2001:11:194:101::1"} },
{ "clusterNetwork": "sriov-oranc", "ip6": "none" },
{ "clusterNetwork": "sriov-oranu", "ip6": "none" },
{ "clusterNetwork": "sriov-oranm", "ip6": "none" }
]
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/gnb12svt-1200du1-546d865594
Containers:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <unknown> default-scheduler 0/2 nodes are available: 1 Insufficient cpu, 1 Insufficient hugepages-1Gi, 1 Insufficient memory, 2 Insufficient windriver.com/isolcpus.
Normal Scheduled <unknown> default-scheduler Successfully assigned default/gnb12svt-1200du1-546d865594-456l5 to controller-1
Warning FailedCreatePodSandBox 75s kubelet, controller-1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "47f637036d3affcc5c55be2c21c6de2503dcb3b97f151ee3ea0057a5eb066369": CNI network could not be set up: CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::113 is already in use
CNI operation for network:sriov-f1c failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
CNI operation for network:sriov-oranm failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
Warning FailedCreatePodSandBox 70s kubelet, controller-1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a2f44bd121b4c8b62809d585f0a5e2f627a0f7a8a5046658fd968296db161897": CNI network could not be set up: CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::113 is already in use
CNI operation for network:sriov-f1c failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
CNI operation for network:sriov-oranm failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
Warning FailedCreatePodSandBox 64s kubelet, controller-1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "797a7434a10875b43dea3cec1bc2d0aa3454161fd183f88e450d03ee8826d055": CNI network could not be set up: CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::113 is already in use
CNI operation for network:sriov-f1c failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
CNI operation for network:sriov-oranm failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
Warning FailedCreatePodSandBox 59s kubelet, controller-1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0f476e3419764fa35ee116f916077ebcf127180334fbac106e71425497e1b6b6": CNI network could not be set up: CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::113 is already in use
CNI operation for network:sriov-f1c failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
CNI operation for network:sriov-oranm failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
Warning FailedCreatePodSandBox 53s kubelet, controller-1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "555dec0dabcd52941ad077dadc29a1f90afb365ffdc3fd46aac0feda8916c525": CNI network could not be set up: CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::113 is already in use
CNI operation for network:sriov-f1c failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
CNI operation for network:sriov-oranm failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
Warning FailedCreatePodSandBox 48s kubelet, controller-1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "010f5bf6b355c08f74baf9d3cd815e2e4595161c208bdb16fb3e0ce549c519ed": CNI network could not be set up: CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::113 is already in use
CNI operation for network:sriov-f1c failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
CNI operation for network:sriov-oranm failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
Warning FailedCreatePodSandBox 42s kubelet, controller-1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1e97d9baded639acaabf547c04b1afae5d7fe92fc1e1a231b378fc86566d7825": CNI network could not be set up: CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::113 is already in use
CNI operation for network:sriov-f1c failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
CNI operation for network:sriov-oranm failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
Warning FailedCreatePodSandBox 37s kubelet, controller-1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f32e9b930ca8255ca00de2336cbaa45c6ad4b5f74ce4aa9bf7f8e3f828a93896": CNI network could not be set up: CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::113 is already in use
CNI operation for network:sriov-f1c failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
CNI operation for network:sriov-oranm failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
Warning FailedCreatePodSandBox 32s kubelet, controller-1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9148e94e0855f22f002d4e02c8574be3e1c2593af6d6e7ee80ad417a0de000b3": CNI network could not be set up: CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::113 is already in use
CNI operation for network:sriov-f1c failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
CNI operation for network:sriov-oranm failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
Warning FailedCreatePodSandBox 3s (x5 over 26s) kubelet, controller-1 (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "59317b3e97f910741e4f8501028be9f8443d0ef4e4866daee2ae7ec52a2151f5": CNI network could not be set up: CNI operation for network:sriov-o1c failed with:IP address reservation failed for network:sriov-o1c with error:failed to allocate IP address for network:sriov-o1c with error:static IP allocation failed, requested IP address:2001:11:194:101::113 is already in use
CNI operation for network:sriov-f1c failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
CNI operation for network:sriov-oranm failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
Please find the logs of danm-cleaner pod.
controller-0:~/sriram$ kubectl get pods -n kube-system -o wide | grep -i cleaner
danm-cleaner-4gbzf 1/1 Running 0 34m 172.16.166.185 controller-1 <none> <none>
danm-cleaner-s7zxz 1/1 Running 0 34m 172.16.192.95 controller-0 <none> <none>
danm-cleaner running in controller-1
controller-0:~/sriram$ kubectl logs -f danm-cleaner-4gbzf -n kube-system
W0630 08:00:07.988368 1 client_config.go:560] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0630 08:00:07.989855 1 leaderelection.go:242] attempting to acquire leader lease kube-system/danm-cleaner...
danm-cleaner running in controller-0
kubectl logs -f danm-cleaner-s7zxz -n kube-system
ould not be cleaned because of error: unable to release ipv4 IP because: no releaseIP Service selected
2021/06/30 08:35:41 INFO: Cleaner freeing IPs belonging to interface:netoranu of Pod:gnb12svt-1200du1-568785cdbb-gqw4h
2021/06/30 08:35:42 WARNING: Danmep 'f5c4ab92-2e1d-4e0b-988f-dd738528f646' in namespace 'default' with network type 'sriov' could not be cleaned because of error: unable to release ipv4 IP because: no releaseIP Service selected
2021/06/30 08:35:46 INFO: Cleaner freeing IPs belonging to interface:netf1c of Pod:gnb12svt-1200du1-568785cdbb-gqw4h
2021/06/30 08:35:47 WARNING: Danmep '0023eea8-f0aa-49ff-8426-154c65d3e683' in namespace 'default' with network type 'sriov' could not be cleaned because of error: unable to release ipv4 IP because: no releaseIP Service selected
2021/06/30 08:35:47 INFO: Cleaner freeing IPs belonging to interface:netdpdk of Pod:gnb12svt-1200du1-568785cdbb-gqw4h
2021/06/30 08:35:48 WARNING: Danmep '14f85401-9bcd-4e63-be31-b0e62d82309b' in namespace 'default' with network type 'sriov' could not be cleaned because of error: unable to release ipv4 IP because: no releaseIP Service selected
2021/06/30 08:35:48 INFO: Cleaner freeing IPs belonging to interface:netoranm of Pod:gnb12svt-1200du1-568785cdbb-gqw4h
2021/06/30 08:35:49 WARNING: Danmep '1f74b70d-f8ce-42f8-9da6-df5b2756f52d' in namespace 'default' with network type 'sriov' could not be cleaned because of error: unable to release ipv4 IP because: no releaseIP Service selected
2021/06/30 08:35:49 INFO: Cleaner freeing IPs belonging to interface:eth0 of Pod:gnb12svt-1200du1-568785cdbb-gqw4h
2021/06/30 08:35:50 release calico managed IP: /usr/local/bin/calicoctl ipam release --ip=172.16.166.148
2021/06/30 08:35:50 WARNING: Danmep '65f8d3fd-eb75-4165-aecb-d636b099a50f' in namespace 'default' with network type 'calico' could not be cleaned because of error: unable to release ipv4 IP because: could not release calico managed IP 172.16.166.148/32, because: exit status 1 | output: no etcd endpoints specified
2021/06/30 08:35:50 INFO: Cleaner freeing IPs belonging to interface:netoranc of Pod:gnb12svt-1200du1-568785cdbb-gqw4h
2021/06/30 08:35:51 WARNING: Danmep 'd09f4745-99a2-4bfe-a1ab-24e516348859' in namespace 'default' with network type 'sriov' could not be cleaned because of error: unable to release ipv4 IP because: no releaseIP Service selected
2021/06/30 08:35:51 INFO: Cleaner freeing IPs belonging to interface:netoranu of Pod:gnb12svt-1200du1-568785cdbb-gqw4h
2021/06/30 08:35:52 WARNING: Danmep 'f5c4ab92-2e1d-4e0b-988f-dd738528f646' in namespace 'default' with network type 'sriov' could not be cleaned because of error: unable to release ipv4 IP because: no releaseIP Service selected
danmeps details
controller-0:~/sriram$ kubectl get danmeps
NAME AGE
0023eea8-f0aa-49ff-8426-154c65d3e683 18h
0111fde6-1e7f-4610-9a3b-3d432b63243e 1s
039b12ac-5cbf-48ea-8481-3d514132a43a 103m
0c997da5-ad23-468e-98f0-7f99fc80a099 2s
14f85401-9bcd-4e63-be31-b0e62d82309b 18h
1876d33a-858a-4215-abb0-f53380253577 110m
1f74b70d-f8ce-42f8-9da6-df5b2756f52d 18h
33ab3f83-7e3b-44bb-98b1-c332ceddd40b 3s
3ba920cf-f2fa-4298-9c3c-877ade58ac08 3s
51d815d4-739f-4107-8efb-8750a017e581 110m
55b5c289-59d2-45ca-9685-e2518901cd2a 110m
65f8d3fd-eb75-4165-aecb-d636b099a50f 18h
6d25ac80-f731-49d5-934a-1ec02b9e98b2 103m
6edbb834-0b69-46a1-9c9d-1cb262edae04 103m
8978875f-c0b2-4d53-a90f-f165d84f81dc 2s
9d898acd-64fe-4fd1-8427-c1e7ea7a4515 110m
d09f4745-99a2-4bfe-a1ab-24e516348859 18h
d9e528e3-cdff-438f-84f0-23e474a2aadd 103m
e2553aa9-3fa4-4594-b714-51a64f0470e6 110m
f5c4ab92-2e1d-4e0b-988f-dd738528f646 18h
controller-0:~/sriram$ kubectl describe danmeps.danm.k8s.io
Name: 0023eea8-f0aa-49ff-8426-154c65d3e683
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200du1
app.kubernetes.io/name=cs-gnb-du
pod-template-hash=568785cdbb
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-29T14:00:18Z
Generation: 1
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:DeviceID:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-29T14:00:18Z
Resource Version: 1816393
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/0023eea8-f0aa-49ff-8426-154c65d3e683
UID: 96848010-9062-4a45-983e-5b46fd427f8d
Spec:
CID: b8ed522bc5b13cad5c04f5b6f2868d4e501b45bc690f903f8fd2530ea3d414e7
Endpoint ID: 0023eea8-f0aa-49ff-8426-154c65d3e683
Host: controller-1
Interface:
Address:
AddressIPv6: 2001:4000:aa:112::2/64
Device ID: 0000:b3:04.4
Mac Address: 00:00:00:00:00:00
Name: netf1c
Proutes: <nil>
proutes6: <nil>
Network Name: sriov-f1c
Network Type: sriov
Pod: gnb12svt-1200du1-568785cdbb-gqw4h
Pod UID: c4026765-cc14-40f5-916c-90d2652f9e5b
API Type: ClusterNetwork
Netns: /var/run/netns/cni-6f4b7d81-fc6c-d0cc-4374-d4fc44912c7d
Events: <none>
Name: 039b12ac-5cbf-48ea-8481-3d514132a43a
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200cuup1
app.kubernetes.io/name=cs-gnb-cu-up
pod-template-hash=5cdd97d9db
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-30T06:53:19Z
Generation: 1
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:DeviceID:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
.:
f:2001:11:195:138::/64:
f:2001:15:208:101::/64:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-30T06:53:19Z
Resource Version: 2127294
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/039b12ac-5cbf-48ea-8481-3d514132a43a
UID: 830e4e64-29bf-4f1d-a67e-d319eb595fbd
Spec:
CID: 644a0f428951deab4c6d61f1a6af22287c05b2b48854ef689e09f64d47777b17
Endpoint ID: 039b12ac-5cbf-48ea-8481-3d514132a43a
Host: controller-0
Interface:
Address:
AddressIPv6: 2001:11:195:101::116/64
Device ID: 0000:b3:04.6
Mac Address: 00:00:00:00:00:00
Name: netdpdk
Proutes: <nil>
proutes6:
2001:11:195:138::/64: 2001:11:195:101::1
2001:15:208:101::/64: 2001:11:195:101::1
Network Name: sriov-dpdk
Network Type: sriov
Pod: gnb12svt-1200cuup1-5cdd97d9db-6m48g
Pod UID: c8422444-755a-4c84-a13d-6830ca06ddb5
API Type: ClusterNetwork
Netns: /var/run/netns/cni-b225dcbb-713d-d991-d2fb-8091fd99a841
Events: <none>
Name: 0619cf49-ee96-4f15-9b48-2c9637e6db6a
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200du1
app.kubernetes.io/name=cs-gnb-du
pod-template-hash=cdb949f4
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-30T08:36:57Z
Generation: 1
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:DeviceID:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-30T08:36:57Z
Resource Version: 2187983
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/0619cf49-ee96-4f15-9b48-2c9637e6db6a
UID: fed67c12-17bc-4811-a8ad-21ea06041a94
Spec:
CID: 96c283d916e1198b138a6ac42539f41af4578561075086680f10b20f9a06b65d
Endpoint ID: 0619cf49-ee96-4f15-9b48-2c9637e6db6a
Host: controller-1
Interface:
Address:
AddressIPv6: none
Device ID: 0000:b3:06.1
Mac Address: 00:00:00:00:00:00
Name: netoranm
Proutes: <nil>
proutes6: <nil>
Network Name: sriov-oranm
Network Type: sriov
Pod: gnb12svt-1200du1-cdb949f4-mxvnn
Pod UID: 4b245673-4232-4f65-bca5-f13cd6bfb600
API Type: ClusterNetwork
Netns: /var/run/netns/cni-96f9827e-b192-8adc-2b57-08905a8dfaeb
Events: <none>
Name: 14f85401-9bcd-4e63-be31-b0e62d82309b
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200du1
app.kubernetes.io/name=cs-gnb-du
pod-template-hash=568785cdbb
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-29T14:00:18Z
Generation: 1
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:DeviceID:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-29T14:00:18Z
Resource Version: 1816386
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/14f85401-9bcd-4e63-be31-b0e62d82309b
UID: cc9ab1ef-0a58-48dc-89c3-bba03920eb9c
Spec:
CID: b8ed522bc5b13cad5c04f5b6f2868d4e501b45bc690f903f8fd2530ea3d414e7
Endpoint ID: 14f85401-9bcd-4e63-be31-b0e62d82309b
Host: controller-1
Interface:
Address:
AddressIPv6: 2001:11:195:101::117/64
Device ID: 0000:b3:05.4
Mac Address: 00:00:00:00:00:00
Name: netdpdk
Proutes: <nil>
proutes6: <nil>
Network Name: sriov-dpdk
Network Type: sriov
Pod: gnb12svt-1200du1-568785cdbb-gqw4h
Pod UID: c4026765-cc14-40f5-916c-90d2652f9e5b
API Type: ClusterNetwork
Netns: /var/run/netns/cni-6f4b7d81-fc6c-d0cc-4374-d4fc44912c7d
Events: <none>
Name: 1876d33a-858a-4215-abb0-f53380253577
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200cucp1
app.kubernetes.io/name=cs-gnb-cu-cp
pod-template-hash=79dd644f6d
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-30T06:46:18Z
Generation: 2
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-30T06:46:19Z
Resource Version: 2125103
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/1876d33a-858a-4215-abb0-f53380253577
UID: 27bcf768-ccfd-49b9-87ba-ddcace0da803
Spec:
CID: d79592a14f9f1990be143d9168dc517ffbe1a8e76b7115a6f241be7908c08271
Endpoint ID: 1876d33a-858a-4215-abb0-f53380253577
Host: controller-0
Interface:
Address: 172.16.192.69/32
AddressIPv6:
Mac Address:
Name: eth0
Proutes: <nil>
proutes6: <nil>
Network Name: default
Network Type: calico
Pod: gnb12svt-1200cucp1-79dd644f6d-fv2bw
Pod UID: 478fe4b2-3993-4f3a-8fb4-a222ed034ebc
API Type: ClusterNetwork
Netns: /var/run/netns/cni-8ae52a1b-66c6-49ec-13dc-4fc72797e6e3
Events: <none>
Name: 1f74b70d-f8ce-42f8-9da6-df5b2756f52d
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200du1
app.kubernetes.io/name=cs-gnb-du
pod-template-hash=568785cdbb
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-29T14:00:21Z
Generation: 1
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:DeviceID:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-29T14:00:21Z
Resource Version: 1816408
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/1f74b70d-f8ce-42f8-9da6-df5b2756f52d
UID: 5c12f8b8-493d-4901-b3f7-79d447a604d2
Spec:
CID: b8ed522bc5b13cad5c04f5b6f2868d4e501b45bc690f903f8fd2530ea3d414e7
Endpoint ID: 1f74b70d-f8ce-42f8-9da6-df5b2756f52d
Host: controller-1
Interface:
Address:
AddressIPv6: none
Device ID: 0000:b3:05.1
Mac Address: 00:00:00:00:00:00
Name: netoranm
Proutes: <nil>
proutes6: <nil>
Network Name: sriov-oranm
Network Type: sriov
Pod: gnb12svt-1200du1-568785cdbb-gqw4h
Pod UID: c4026765-cc14-40f5-916c-90d2652f9e5b
API Type: ClusterNetwork
Netns: /var/run/netns/cni-6f4b7d81-fc6c-d0cc-4374-d4fc44912c7d
Events: <none>
Name: 4f2a9950-85c8-413e-b4c1-428c748b3fed
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200du1
app.kubernetes.io/name=cs-gnb-du
pod-template-hash=cdb949f4
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-30T08:36:55Z
Generation: 1
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:DeviceID:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-30T08:36:55Z
Resource Version: 2187971
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/4f2a9950-85c8-413e-b4c1-428c748b3fed
UID: 142f393e-7677-4410-b6bc-b2054ab2919c
Spec:
CID: 96c283d916e1198b138a6ac42539f41af4578561075086680f10b20f9a06b65d
Endpoint ID: 4f2a9950-85c8-413e-b4c1-428c748b3fed
Host: controller-1
Interface:
Address:
AddressIPv6: none
Device ID: 0000:b3:04.5
Mac Address: 00:00:00:00:00:00
Name: netoranc
Proutes: <nil>
proutes6: <nil>
Network Name: sriov-oranc
Network Type: sriov
Pod: gnb12svt-1200du1-cdb949f4-mxvnn
Pod UID: 4b245673-4232-4f65-bca5-f13cd6bfb600
API Type: ClusterNetwork
Netns: /var/run/netns/cni-96f9827e-b192-8adc-2b57-08905a8dfaeb
Events: <none>
Name: 51d815d4-739f-4107-8efb-8750a017e581
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200cucp1
app.kubernetes.io/name=cs-gnb-cu-cp
pod-template-hash=79dd644f6d
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-30T06:46:20Z
Generation: 1
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:DeviceID:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-30T06:46:20Z
Resource Version: 2125108
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/51d815d4-739f-4107-8efb-8750a017e581
UID: 4e952408-4a67-4056-9f83-2b4951c60315
Spec:
CID: d79592a14f9f1990be143d9168dc517ffbe1a8e76b7115a6f241be7908c08271
Endpoint ID: 51d815d4-739f-4107-8efb-8750a017e581
Host: controller-0
Interface:
Address:
AddressIPv6: 2001:4000:aa:114::1/64
Device ID: 0000:b3:06.1
Mac Address: 00:00:00:00:00:00
Name: nete1c
Proutes: <nil>
proutes6: <nil>
Network Name: sriov-e1c
Network Type: sriov
Pod: gnb12svt-1200cucp1-79dd644f6d-fv2bw
Pod UID: 478fe4b2-3993-4f3a-8fb4-a222ed034ebc
API Type: ClusterNetwork
Netns: /var/run/netns/cni-8ae52a1b-66c6-49ec-13dc-4fc72797e6e3
Events: <none>
Name: 55b5c289-59d2-45ca-9685-e2518901cd2a
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200cucp1
app.kubernetes.io/name=cs-gnb-cu-cp
pod-template-hash=79dd644f6d
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-30T06:46:19Z
Generation: 1
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:DeviceID:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-30T06:46:19Z
Resource Version: 2125104
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/55b5c289-59d2-45ca-9685-e2518901cd2a
UID: 5ff96838-f2f2-48ec-92ab-6c4348930299
Spec:
CID: d79592a14f9f1990be143d9168dc517ffbe1a8e76b7115a6f241be7908c08271
Endpoint ID: 55b5c289-59d2-45ca-9685-e2518901cd2a
Host: controller-0
Interface:
Address:
AddressIPv6: 2001:4000:aa:112::1/64
Device ID: 0000:b3:05.6
Mac Address: 00:00:00:00:00:00
Name: netf1c
Proutes: <nil>
proutes6: <nil>
Network Name: sriov-f1c
Network Type: sriov
Pod: gnb12svt-1200cucp1-79dd644f6d-fv2bw
Pod UID: 478fe4b2-3993-4f3a-8fb4-a222ed034ebc
API Type: ClusterNetwork
Netns: /var/run/netns/cni-8ae52a1b-66c6-49ec-13dc-4fc72797e6e3
Events: <none>
Name: 58a2becc-7ebb-4237-81a1-c5518011f5dc
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200du1
app.kubernetes.io/name=cs-gnb-du
pod-template-hash=cdb949f4
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-30T08:36:56Z
Generation: 1
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:DeviceID:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-30T08:36:56Z
Resource Version: 2187973
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/58a2becc-7ebb-4237-81a1-c5518011f5dc
UID: 991c62a6-a72e-4ae7-9944-fcc90de6f75c
Spec:
CID: 96c283d916e1198b138a6ac42539f41af4578561075086680f10b20f9a06b65d
Endpoint ID: 58a2becc-7ebb-4237-81a1-c5518011f5dc
Host: controller-1
Interface:
Address:
AddressIPv6: none
Device ID: 0000:b3:05.3
Mac Address: 00:00:00:00:00:00
Name: netoranu
Proutes: <nil>
proutes6: <nil>
Network Name: sriov-oranu
Network Type: sriov
Pod: gnb12svt-1200du1-cdb949f4-mxvnn
Pod UID: 4b245673-4232-4f65-bca5-f13cd6bfb600
API Type: ClusterNetwork
Netns: /var/run/netns/cni-96f9827e-b192-8adc-2b57-08905a8dfaeb
Events: <none>
Name: 65f8d3fd-eb75-4165-aecb-d636b099a50f
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200du1
app.kubernetes.io/name=cs-gnb-du
pod-template-hash=568785cdbb
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-29T14:00:18Z
Generation: 2
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-29T14:00:19Z
Resource Version: 1816395
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/65f8d3fd-eb75-4165-aecb-d636b099a50f
UID: 352507ed-a109-4eda-80b4-87a87b644517
Spec:
CID: b8ed522bc5b13cad5c04f5b6f2868d4e501b45bc690f903f8fd2530ea3d414e7
Endpoint ID: 65f8d3fd-eb75-4165-aecb-d636b099a50f
Host: controller-1
Interface:
Address: 172.16.166.148/32
AddressIPv6:
Mac Address:
Name: eth0
Proutes: <nil>
proutes6: <nil>
Network Name: default
Network Type: calico
Pod: gnb12svt-1200du1-568785cdbb-gqw4h
Pod UID: c4026765-cc14-40f5-916c-90d2652f9e5b
API Type: ClusterNetwork
Netns: /var/run/netns/cni-6f4b7d81-fc6c-d0cc-4374-d4fc44912c7d
Events: <none>
Name: 6d25ac80-f731-49d5-934a-1ec02b9e98b2
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200cuup1
app.kubernetes.io/name=cs-gnb-cu-up
pod-template-hash=5cdd97d9db
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-30T06:53:19Z
Generation: 1
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:DeviceID:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-30T06:53:19Z
Resource Version: 2127304
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/6d25ac80-f731-49d5-934a-1ec02b9e98b2
UID: 0191a644-6073-444f-a279-fb0917ae65c3
Spec:
CID: 644a0f428951deab4c6d61f1a6af22287c05b2b48854ef689e09f64d47777b17
Endpoint ID: 6d25ac80-f731-49d5-934a-1ec02b9e98b2
Host: controller-0
Interface:
Address:
AddressIPv6: 2001:4000:aa:114::2/64
Device ID: 0000:b3:05.3
Mac Address: 00:00:00:00:00:00
Name: nete1c
Proutes: <nil>
proutes6: <nil>
Network Name: sriov-e1c
Network Type: sriov
Pod: gnb12svt-1200cuup1-5cdd97d9db-6m48g
Pod UID: c8422444-755a-4c84-a13d-6830ca06ddb5
API Type: ClusterNetwork
Netns: /var/run/netns/cni-b225dcbb-713d-d991-d2fb-8091fd99a841
Events: <none>
Name: 6edbb834-0b69-46a1-9c9d-1cb262edae04
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200cuup1
app.kubernetes.io/name=cs-gnb-cu-up
pod-template-hash=5cdd97d9db
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-30T06:53:18Z
Generation: 2
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-30T06:53:19Z
Resource Version: 2127306
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/6edbb834-0b69-46a1-9c9d-1cb262edae04
UID: 82f9f703-58b4-4e95-b470-1ece34a61288
Spec:
CID: 644a0f428951deab4c6d61f1a6af22287c05b2b48854ef689e09f64d47777b17
Endpoint ID: 6edbb834-0b69-46a1-9c9d-1cb262edae04
Host: controller-0
Interface:
Address: 172.16.192.65/32
AddressIPv6:
Mac Address:
Name: eth0
Proutes: <nil>
proutes6: <nil>
Network Name: default
Network Type: calico
Pod: gnb12svt-1200cuup1-5cdd97d9db-6m48g
Pod UID: c8422444-755a-4c84-a13d-6830ca06ddb5
API Type: ClusterNetwork
Netns: /var/run/netns/cni-b225dcbb-713d-d991-d2fb-8091fd99a841
Events: <none>
Name: 9d898acd-64fe-4fd1-8427-c1e7ea7a4515
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200cucp1
app.kubernetes.io/name=cs-gnb-cu-cp
pod-template-hash=79dd644f6d
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-30T06:46:20Z
Generation: 1
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:DeviceID:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
.:
f:2001:31:212:2::/64:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-30T06:46:20Z
Resource Version: 2125113
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/9d898acd-64fe-4fd1-8427-c1e7ea7a4515
UID: 3bb17388-fbe2-4380-8752-3fa438332d01
Spec:
CID: d79592a14f9f1990be143d9168dc517ffbe1a8e76b7115a6f241be7908c08271
Endpoint ID: 9d898acd-64fe-4fd1-8427-c1e7ea7a4515
Host: controller-0
Interface:
Address:
AddressIPv6: 2001:11:194:101::111/64
Device ID: 0000:b3:06.5
Mac Address: 00:00:00:00:00:00
Name: neto1c
Proutes: <nil>
proutes6:
2001:31:212:2::/64: 2001:11:194:101::1
Network Name: sriov-o1c
Network Type: sriov
Pod: gnb12svt-1200cucp1-79dd644f6d-fv2bw
Pod UID: 478fe4b2-3993-4f3a-8fb4-a222ed034ebc
API Type: ClusterNetwork
Netns: /var/run/netns/cni-8ae52a1b-66c6-49ec-13dc-4fc72797e6e3
Events: <none>
Name: d09f4745-99a2-4bfe-a1ab-24e516348859
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200du1
app.kubernetes.io/name=cs-gnb-du
pod-template-hash=568785cdbb
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-29T14:00:19Z
Generation: 1
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:DeviceID:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-29T14:00:19Z
Resource Version: 1816400
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/d09f4745-99a2-4bfe-a1ab-24e516348859
UID: 6020e033-6c06-49b0-8ab7-be3ec0d4d421
Spec:
CID: b8ed522bc5b13cad5c04f5b6f2868d4e501b45bc690f903f8fd2530ea3d414e7
Endpoint ID: d09f4745-99a2-4bfe-a1ab-24e516348859
Host: controller-1
Interface:
Address:
AddressIPv6: none
Device ID: 0000:b3:05.5
Mac Address: 00:00:00:00:00:00
Name: netoranc
Proutes: <nil>
proutes6: <nil>
Network Name: sriov-oranc
Network Type: sriov
Pod: gnb12svt-1200du1-568785cdbb-gqw4h
Pod UID: c4026765-cc14-40f5-916c-90d2652f9e5b
API Type: ClusterNetwork
Netns: /var/run/netns/cni-6f4b7d81-fc6c-d0cc-4374-d4fc44912c7d
Events: <none>
Name: d9e528e3-cdff-438f-84f0-23e474a2aadd
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200cuup1
app.kubernetes.io/name=cs-gnb-cu-up
pod-template-hash=5cdd97d9db
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-30T06:53:20Z
Generation: 1
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:DeviceID:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
.:
f:2001:31:212:2::/64:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-30T06:53:20Z
Resource Version: 2127308
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/d9e528e3-cdff-438f-84f0-23e474a2aadd
UID: a5dc14d4-5600-4c95-9f3d-c89a0508f875
Spec:
CID: 644a0f428951deab4c6d61f1a6af22287c05b2b48854ef689e09f64d47777b17
Endpoint ID: d9e528e3-cdff-438f-84f0-23e474a2aadd
Host: controller-0
Interface:
Address:
AddressIPv6: 2001:11:194:101::112/64
Device ID: 0000:b3:05.4
Mac Address: 00:00:00:00:00:00
Name: neto1c
Proutes: <nil>
proutes6:
2001:31:212:2::/64: 2001:11:194:101::1
Network Name: sriov-o1c
Network Type: sriov
Pod: gnb12svt-1200cuup1-5cdd97d9db-6m48g
Pod UID: c8422444-755a-4c84-a13d-6830ca06ddb5
API Type: ClusterNetwork
Netns: /var/run/netns/cni-b225dcbb-713d-d991-d2fb-8091fd99a841
Events: <none>
Name: e2553aa9-3fa4-4594-b714-51a64f0470e6
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200cucp1
app.kubernetes.io/name=cs-gnb-cu-cp
pod-template-hash=79dd644f6d
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-30T06:46:18Z
Generation: 1
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:DeviceID:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
.:
f:2001:11:195:138::/64:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-30T06:46:18Z
Resource Version: 2125092
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/e2553aa9-3fa4-4594-b714-51a64f0470e6
UID: 4629766a-315a-4e80-b694-1dcbc770c1a0
Spec:
CID: d79592a14f9f1990be143d9168dc517ffbe1a8e76b7115a6f241be7908c08271
Endpoint ID: e2553aa9-3fa4-4594-b714-51a64f0470e6
Host: controller-0
Interface:
Address:
AddressIPv6: 2001:11:195:101::111/64
Device ID: 0000:b3:06.4
Mac Address: 00:00:00:00:00:00
Name: netx2c
Proutes: <nil>
proutes6:
2001:11:195:138::/64: 2001:11:195:101::1
Network Name: sriov-x2c
Network Type: sriov
Pod: gnb12svt-1200cucp1-79dd644f6d-fv2bw
Pod UID: 478fe4b2-3993-4f3a-8fb4-a222ed034ebc
API Type: ClusterNetwork
Netns: /var/run/netns/cni-8ae52a1b-66c6-49ec-13dc-4fc72797e6e3
Events: <none>
Name: e3e6e439-1ff1-4ec2-ad6b-06a751570192
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200du1
app.kubernetes.io/name=cs-gnb-du
pod-template-hash=cdb949f4
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-30T08:36:54Z
Generation: 2
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-30T08:36:55Z
Resource Version: 2187965
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/e3e6e439-1ff1-4ec2-ad6b-06a751570192
UID: 2fdbd851-1cc3-40a7-9281-44d9b0fb9eea
Spec:
CID: 96c283d916e1198b138a6ac42539f41af4578561075086680f10b20f9a06b65d
Endpoint ID: e3e6e439-1ff1-4ec2-ad6b-06a751570192
Host: controller-1
Interface:
Address: 172.16.166.183/32
AddressIPv6:
Mac Address:
Name: eth0
Proutes: <nil>
proutes6: <nil>
Network Name: default
Network Type: calico
Pod: gnb12svt-1200du1-cdb949f4-mxvnn
Pod UID: 4b245673-4232-4f65-bca5-f13cd6bfb600
API Type: ClusterNetwork
Netns: /var/run/netns/cni-96f9827e-b192-8adc-2b57-08905a8dfaeb
Events: <none>
Name: e83db804-4a42-4732-abb2-f8f9ed6742a0
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200du1
app.kubernetes.io/name=cs-gnb-du
pod-template-hash=cdb949f4
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-30T08:36:55Z
Generation: 1
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:DeviceID:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-30T08:36:55Z
Resource Version: 2187969
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/e83db804-4a42-4732-abb2-f8f9ed6742a0
UID: 41d928d3-5600-4a0c-9a9e-4b07d09836dc
Spec:
CID: 96c283d916e1198b138a6ac42539f41af4578561075086680f10b20f9a06b65d
Endpoint ID: e83db804-4a42-4732-abb2-f8f9ed6742a0
Host: controller-1
Interface:
Address:
AddressIPv6: 2001:4000:aa:112::3/64
Device ID: 0000:b3:06.2
Mac Address: 00:00:00:00:00:00
Name: netf1c
Proutes: <nil>
proutes6: <nil>
Network Name: sriov-f1c
Network Type: sriov
Pod: gnb12svt-1200du1-cdb949f4-mxvnn
Pod UID: 4b245673-4232-4f65-bca5-f13cd6bfb600
API Type: ClusterNetwork
Netns: /var/run/netns/cni-96f9827e-b192-8adc-2b57-08905a8dfaeb
Events: <none>
Name: f5c4ab92-2e1d-4e0b-988f-dd738528f646
Namespace: default
Labels: app.kubernetes.io/instance=gnb12svt-1200du1
app.kubernetes.io/name=cs-gnb-du
pod-template-hash=568785cdbb
Annotations: <none>
API Version: danm.k8s.io/v1
Kind: DanmEp
Metadata:
Creation Timestamp: 2021-06-29T14:00:20Z
Generation: 1
Managed Fields:
API Version: danm.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:pod-template-hash:
f:spec:
.:
f:CID:
f:EndpointID:
f:Host:
f:Interface:
.:
f:Address:
f:AddressIPv6:
f:DeviceID:
f:MacAddress:
f:Name:
f:proutes:
f:proutes6:
f:NetworkName:
f:NetworkType:
f:Pod:
f:PodUID:
f:apiType:
f:netns:
Manager: danm
Operation: Update
Time: 2021-06-29T14:00:20Z
Resource Version: 1816407
Self Link: /apis/danm.k8s.io/v1/namespaces/default/danmeps/f5c4ab92-2e1d-4e0b-988f-dd738528f646
UID: 360081a4-fccb-4a08-bb3e-bf285ae5d647
Spec:
CID: b8ed522bc5b13cad5c04f5b6f2868d4e501b45bc690f903f8fd2530ea3d414e7
Endpoint ID: f5c4ab92-2e1d-4e0b-988f-dd738528f646
Host: controller-1
Interface:
Address:
AddressIPv6: none
Device ID: 0000:b3:05.2
Mac Address: 00:00:00:00:00:00
Name: netoranu
Proutes: <nil>
proutes6: <nil>
Network Name: sriov-oranu
Network Type: sriov
Pod: gnb12svt-1200du1-568785cdbb-gqw4h
Pod UID: c4026765-cc14-40f5-916c-90d2652f9e5b
API Type: ClusterNetwork
Netns: /var/run/netns/cni-6f4b7d81-fc6c-d0cc-4374-d4fc44912c7d
Events: <none>
@eMGabriel "2021/06/30 08:35:47 WARNING: Danmep '0023eea8-f0aa-49ff-8426-154c65d3e683' in namespace 'default' with network type 'sriov' could not be cleaned because of error: unable to release ipv4 IP because: no releaseIP Service selected" maybe the result of your refactoring? seems like Cleaner can't handle SR-IOV type of interfaces. I'm guessing it doesn't recognize it as dnyamic DANM type
@eMGabriel I think the problem is that the Cleaner Service Farmework implementation assumes DanmEps always have a V4 address. so it 100% tries to find an implementation for both V4 and V6, and treats it is an error when the DANM API returns a "no" for an empty address V4 and V6 address cleaning should be independent from each other, and not releasing an empty address should not be treated as an error
@sriramec hopefully following PR addresses your issue: https://github.com/nokia/danm-utils/pull/30 could you try it out in your environment?
in case any issues, let's move the discussion under that PR!
We observed one ip leak issue in one of our systems. To solve the ip leak, that I deployed cleaner pods.(with the fix suggested in the PR). Reinstalled the other pod, although ip leak is not seen. We see these messages and pod is not getting deployed.
Warning FailedCreatePodSandBox 32s kubelet, controller-1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "bb4e568f406cd516ec765d00eb5eb360159e6cf83a48722c9a7eedb959c4ff3c": CNI network could not be set up: CNI operation for network:sriov-oranu failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message CNI operation for network:sriov-o1c-host1 failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message Warning FailedCreatePodSandBox 26s kubelet, controller-1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1110c5dfa4cccb82bc044533ec134825f1db5f4305cb4bdd96c49a6d2522ba94": CNI network could not be set up: CNI operation for network:sriov-oranu failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message CNI operation for network:sriov-o1c-host1 failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message Warning FailedCreatePodSandBox 20s kubelet, controller-1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "770ebc74a4a39765a2940306c1e8fca2f5189afb9ca398a6cc9f055e76389077": CNI network could not be set up: CNI operation for network:sriov-oranu failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message CNI operation for network:sriov-o1c-host1 failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message Warning FailedCreatePodSandBox 2s (x3 over 14s) kubelet, controller-1 (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f7d6ce70c309155e63ec31ffb6bac487f78fc9f4203b85354b070aecac8e96e9": CNI network could not be set up: CNI operation for network:sriov-oranu failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message CNI operation for network:sriov-o1c-host1 failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
Please let me know if further information is required
CNI operation for network:sriov-o1c-host1 failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call failed:netplugin failed with no error message
that part is not thrown by DANM. you need to look for why the invoked CNI plugin returns error to us
One of the pods which uses sriov interfaces configured using DANM meta CNI is stuck in container creating state
controller-0:~# kubectl get pods -n csblrlabsvtcc1ec52-1052
NAME READY STATUS RESTARTS AGE
csblrlabsvtcc1ec52-1052cucp1-f9f4465d4-zqmlb 0/9 ContainerCreating 0 4h20m
If I do describe of that pod I m seeing like below,
kubectl describe pod csblrlabsvtcc1ec52-1052cucp1-f9f4465d4-zqmlb -n csblrlabsvtcc1ec52-1052
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned csblrlabsvtcc1ec52-1052/csblrlabsvtcc1ec52-1052cucp1-f9f4465d4-zqmlb to controller-0
Warning FailedCreatePodSandBox 4h10m kubelet, controller-0 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "75eb768e8dc0742a2546596e9732e8379c5eeba7531348c0a8ae0c0ba4d8fd51": CNI network could not be set up: CNI operation for network:sriov-e1c failed with:IP address reservation failed for network:sriov-e1c with error:failed to allocate IP address for network:sriov-e1c with error:static IP allocation failed, requested IP address:2001:4000:aa:34::1 is already in use
CNI operation for network:sriov-o1c-host0 failed with:IP address reservation failed for network:sriov-o1c-host0 with error:failed to allocate IP address for network:sriov-o1c-host0 with error:static IP allocation failed, requested IP address:2005:11:6:1::1 is already in use
Warning FailedCreatePodSandBox 4h9m kubelet, controller-0 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "34f6434a1983d2a0fe05a3e9bcb5dfac52671d378ad402bd69da10caec7e2169": CNI network could not be set up: CNI operation for network:sriov-e1c failed with:IP address reservation failed for network:sriov-e1c with error:failed to allocate IP address for network:sriov-e1c with error:static IP allocation failed, requested IP address:2001:4000:aa:34::1 is already in use
CNI operation for network:sriov-o1c-host0 failed with:IP address reservation failed for network:sriov-o1c-host0 with error:failed to allocate IP address for network:sriov-o1c-host0 with error:static IP allocation failed, requested IP address:2005:11:6:1::1 is already in use
Warning FailedCreatePodSandBox 4h9m kubelet, controller-0 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e4773d03aea09f693dcf655b5c2b32dd89d14a55f0bf750565939c5cbe15d001": CNI network could not be set up: CNI operation for network:sriov-e1c failed with:IP address reservation failed for network:sriov-e1c with error:failed to allocate IP address for network:sriov-e1c with error:static IP allocation failed, requested IP address:2001:4000:aa:34::1 is already in use
CNI operation for network:sriov-o1c-host0 failed with:IP address reservation failed for network:sriov-o1c-host0 with error:failed to allocate IP address for network:sriov-o1c-host0 with error:static IP allocation failed, requested IP address:2005:11:6:1::1 is already in use
Warning FailedCreatePodSandBox 4h9m kubelet, controller-0 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "aff7ffe450f78b37521b8cc7d8b738e4cbc3403962816c605609226e098f1c81": CNI network could not be set up: CNI operation for network:sriov-e1c failed with:IP address reservation failed for network:sriov-e1c with error:failed to allocate IP address for network:sriov-e1c with error:static IP allocation failed, requested IP address:2001:4000:aa:34::1 is already in use
There is ip leak for two Cluster Networks sriov-e1c and sriov-o1c-host0.
I have deployed the danm cleaner pods. In kube-system namespace.
danm-cleaner-jt5gd 1/1 Running 0 4h11m
danm-cleaner-l86ps 1/1 Running 0 4h11m
I have deployed danm-cleaner using this manifest yaml https://github.com/nokia/danm-utils/blob/master/integration/manifests/cleaner/cleaner.yaml
Let me know where could be the problem of ip leak. I didnt see these two ip's(2001:4000:aa:34::1 & 2005:11:6:1::1) getting used in any other pods.
One more question, I have not deployed danm-cleaner for calico(https://github.com/nokia/danm-utils/blob/master/integration/manifests/cleaner/cleaner-for-calico.yaml). Do you think it is necessary to solve this issue, from the documentation I see it is required to solve the problem of calico leaked IPs?
I am attaching the o/p of danmeps
controller-0:~# kubectl get danmeps
NAME AGE
01c1d131-acbb-4965-af84-f31b71085eb5 4d4h
080e08a1-ecd6-4a53-91d0-7307c1e3784b 22h
1ef03916-4398-4131-938f-25c0b5277301 4d4h
28e75b79-ca33-4a17-8b78-869b34949eeb 22h
2c6a9b6f-222e-4b72-bf04-c87cf3ad21a0 22h
401391a7-cc3e-4b02-8017-218633da91bb 4d4h
44803401-e22c-451c-a8b2-a442a6635680 4d4h
680e84fb-b23f-4fba-adcb-499e78d10fac 22h
75732617-bd5e-484f-9377-05b486d5e1bd 22h
7e97d975-8d2b-41bf-a2b6-2be4dbcccb2b 4d4h
8252fdb2-fc0f-4b8c-9645-3f28be0256f5 22h
8aaf0715-73af-4894-86c7-08136d392d60 4d4h
8ede4cca-4396-46e4-bf74-17323b116952 4d4h
9546339d-77fd-4d25-be6c-e1b4ce14e965 22h
9785c533-d1a2-45d9-ae6f-2f82d3267674 4d4h
9e831733-9d4c-4633-a24b-64a844f3ac9f 22h
b7ca7278-1f56-4910-9a1f-9c2a329e00a8 22h
c1758916-8bc5-458e-b6f6-44ebb5322f16 4d4h
c9b4a11a-3a1c-4b45-b61f-56cb19ab0227 4d4h
ed1496cf-326d-481b-bf79-1cb0775f3946 22h
eef7448e-19c8-436b-99a4-b37e1e32799c 22h
controller-0:~# kubectl get danmeps -n csblrlabsvtcc1ec52-1052
NAME AGE
0ae6b04d-0046-4436-a6b4-3491e4875f3c 3s
0bf85cbe-7062-4750-bb20-b228cc4f8d6c 2s
1051e61a-1a98-483a-8992-92d16dae2ae3 22h
158be71c-aac9-47ef-852d-1349ea2c0935 22h
16c7fca1-6823-4a62-bd53-396f625fcacc 22h
2161b832-9465-4fc0-acd6-f9afb121e539 4d4h
226a304b-aceb-4793-a482-ee5231026b04 4d4h
23307630-6918-4102-a616-fd004c0be4cc 22h
35ca5625-fac9-48ef-801c-ab09da012a03 22h
41acd01b-12a7-46dd-8c44-d0be5d9bf696 22h
45c9fb00-d864-4177-89be-51737085252f 4d4h
5206d01f-e49c-41a8-8c5d-48f194920694 4d4h
5e580970-f2da-410e-a876-107233e6a3dc 1s
6d1e7f71-47cb-4c62-b646-8ed6a4e0981a 4d4h
8a511fdc-75b3-4bfa-8602-09eac4d107f3 22h
9e5b260b-057d-492c-b0a7-c5056528447a 3s
9eb25c78-fcd3-4b4b-9094-662e841ee1c9 22h
a9aba4e5-efee-436c-a130-d8ee91d8a808 22h
ea028891-ab23-4bc2-b468-ca35e6ce1f25 4d4h
eb301b2c-8d21-4ae0-a3ac-af4ff35d81e8 22h
f10a4891-115c-48ae-a369-86b9e6905b84 4d4h
f48b5ba1-01b4-4f03-aa19-f842033901fd 22h
Let me know if any information is required ?