sriov-network-device-plugin
sriov-network-device-plugin copied to clipboard
k8snetworkplumbingwg
use usespace vdpa can't find the pfName because the net path is wrong
What happened?
There is no net path in Virtio-net is the net layer of virtio, and the standard kernel driver places this layer separately into the virtio1 path
What did you expect to happen?
If the net folder is not found in the /sys/bus/pci/devices/pciAddr/physfn/net path, I hope you can find the folder starting with virtio again
What are the minimal steps needed to reproduce the bug?
No**/sys/bus/pci/devices/pciAddr/physfn/net** Only the /sys/bus/pci/devices/pciAddr virtioxx/physfn/net
Anything else we need to know?
Component Versions
Please fill in the below table with the version numbers of components used.
| Component | Version |
|---|---|
| SR-IOV Network Device Plugin | 0bf8f4fa9dc4ef6a0b88c875142815676d3bf311 |
| SR-IOV CNI Plugin | 0bf8f4fa9dc4ef6a0b88c875142815676d3bf311 |
| Multus | |
| Kubernetes | v1.24.0 |
| OS | ubuntu 22.04 |
Config Files
Config file locations may be config dependent.
Device pool config file location (Try '/etc/pcidp/config.json')
Multus config (Try '/etc/cni/multus/net.d')
CNI config (Try '/etc/cni/net.d/')
Kubernetes deployment type ( Bare Metal, Kubeadm etc.)
Kubeconfig file
SR-IOV Network Custom Resource Definition
Logs
SR-IOV Network Device Plugin Logs (use kubectl logs $PODNAME)
root@master:~/temp/kubevirt# kubectl logs -f -n kube-system kube-sriov-device-plugin-amd64-d6vsq I1108 14:43:33.442520 1 manager.go:51] Using Kubelet Plugin Registry Mode I1108 14:43:33.442648 1 main.go:44] resource manager reading configs I1108 14:43:33.442685 1 manager.go:79] raw ResourceList: { "resourceList": [{ "resourceName": "sriov_dpu", "resourcePrefix": "yusur.tech", "selectors": { "rootDevices": ["0000:01:00.0#1-6"] } } ] } I1108 14:43:33.442759 1 factory.go:166] net device selector for resource sriov_dpu is &{DeviceSelectors:{Vendors:[] Devices:[] Drivers:[] PciAddresses:[]} PfNames:[] RootDevices:[0000:01:00.0#1-6] LinkTypes:[] DDPProfiles:[] IsRdma:false NeedVhostNet:false} I1108 14:43:33.442781 1 manager.go:99] unmarshalled ResourceList: [{ResourcePrefix:yusur.tech ResourceName:sriov_dpu DeviceType:netDevice Selectors:0xc00000cd38 SelectorObj:0xc0001b32b0}] I1108 14:43:33.442815 1 manager.go:200] validating resource name "yusur.tech/sriov_dpu" I1108 14:43:33.442818 1 main.go:60] Discovering host devices I1108 14:43:33.469296 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.0 02 unknown unknown I1108 14:43:33.469535 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.1 02 unknown unknown I1108 14:43:33.469605 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.2 02 unknown unknown I1108 14:43:33.469659 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.3 02 unknown unknown I1108 14:43:33.469719 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.4 02 unknown unknown I1108 14:43:33.469767 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.5 02 unknown unknown I1108 14:43:33.469814 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.6 02 unknown unknown I1108 14:43:33.469875 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.7 02 unknown unknown I1108 14:43:33.469931 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0b.0 02 unknown unknown I1108 14:43:33.469990 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0c.0 02 Red Hat, Inc. Virtio network device I1108 14:43:33.470009 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.0 02 Red Hat, Inc. Virtio network device I1108 14:43:33.470035 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.1 02 Red Hat, Inc. Virtio network device I1108 14:43:33.470045 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.2 02 Red Hat, Inc. Virtio network device I1108 14:43:33.470054 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.3 02 Red Hat, Inc. Virtio network device I1108 14:43:33.470064 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.4 02 Red Hat, Inc. Virtio network device I1108 14:43:33.470075 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.5 02 Red Hat, Inc. Virtio network device I1108 14:43:33.470086 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.6 02 Red Hat, Inc. Virtio network device I1108 14:43:33.470097 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.7 02 Red Hat, Inc. Virtio network device I1108 14:43:33.470106 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:02:00.0 02 Red Hat, Inc. Virtio network device I1108 14:43:33.470127 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:10:00.0 02 Red Hat, Inc. Virtio network device I1108 14:43:33.470151 1 main.go:66] Initializing resource servers I1108 14:43:33.470157 1 manager.go:105] number of config: 1 I1108 14:43:33.470160 1 manager.go:109] I1108 14:43:33.470162 1 manager.go:110] Creating new ResourcePool: sriov_dpu I1108 14:43:33.470166 1 manager.go:111] DeviceType: netDevice I1108 14:43:33.470339 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.1. error getting devlink device attributes for net device 0000:00:0a.0 no such device I1108 14:43:33.470511 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.2. error getting devlink device attributes for net device 0000:00:0a.0 no such device I1108 14:43:33.470694 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.3. error getting devlink device attributes for net device 0000:00:0a.0 no such device I1108 14:43:33.471309 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.4. error getting devlink device attributes for net device 0000:00:0a.0 no such device I1108 14:43:33.471644 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.5. error getting devlink device attributes for net device 0000:00:0a.0 no such device I1108 14:43:33.471959 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.6. error getting devlink device attributes for net device 0000:00:0a.0 no such device I1108 14:43:33.472276 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.7. error getting devlink device attributes for net device 0000:00:0a.0 no such device I1108 14:43:33.472473 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0b.0. error getting devlink device attributes for net device 0000:00:0b.0 no such device I1108 14:43:33.472583 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0c.0. error getting devlink device attributes for net device 0000:00:0c.0 no such device W1108 14:43:33.472592 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:00:0c.0/net: no such file or directory" I1108 14:43:33.472827 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.1. error getting devlink device attributes for net device 0000:01:00.0 no such device W1108 14:43:33.472837 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.1/net: no such file or directory" I1108 14:43:33.472977 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.2. error getting devlink device attributes for net device 0000:01:00.0 no such device W1108 14:43:33.472984 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.2/net: no such file or directory" I1108 14:43:33.473145 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.3. error getting devlink device attributes for net device 0000:01:00.0 no such device W1108 14:43:33.473154 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.3/net: no such file or directory" I1108 14:43:33.473338 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.4. error getting devlink device attributes for net device 0000:01:00.0 no such device W1108 14:43:33.473347 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.4/net: no such file or directory" I1108 14:43:33.473572 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.5. error getting devlink device attributes for net device 0000:01:00.0 no such device W1108 14:43:33.473580 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.5/net: no such file or directory" I1108 14:43:33.473825 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.6. error getting devlink device attributes for net device 0000:01:00.0 no such device W1108 14:43:33.473834 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.6/net: no such file or directory" I1108 14:43:33.474096 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.7. error getting devlink device attributes for net device 0000:01:00.0 no such device W1108 14:43:33.474104 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.7/net: no such file or directory" I1108 14:43:33.474171 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:02:00.0. error getting devlink device attributes for net device 0000:02:00.0 no such device W1108 14:43:33.474178 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:02:00.0/net: no such file or directory" I1108 14:43:33.474239 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:10:00.0. error getting devlink device attributes for net device 0000:10:00.0 no such device W1108 14:43:33.474246 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:10:00.0/net: no such file or directory" I1108 14:43:33.474261 1 factory.go:106] device added: [pciAddr: 0000:01:00.2, vendor: 1af4, device: 1041, driver: virtio-pci] I1108 14:43:33.474263 1 factory.go:106] device added: [pciAddr: 0000:01:00.3, vendor: 1af4, device: 1041, driver: virtio-pci] I1108 14:43:33.474265 1 factory.go:106] device added: [pciAddr: 0000:01:00.4, vendor: 1af4, device: 1041, driver: virtio-pci] I1108 14:43:33.474267 1 factory.go:106] device added: [pciAddr: 0000:01:00.5, vendor: 1af4, device: 1041, driver: virtio-pci] I1108 14:43:33.474271 1 factory.go:106] device added: [pciAddr: 0000:01:00.6, vendor: 1af4, device: 1041, driver: virtio-pci] I1108 14:43:33.474273 1 factory.go:106] device added: [pciAddr: 0000:01:00.7, vendor: 1af4, device: 1041, driver: virtio-pci] I1108 14:43:33.474281 1 manager.go:139] New resource server is created for sriov_dpu ResourcePool I1108 14:43:33.474285 1 main.go:72] Starting all servers... I1108 14:43:33.474446 1 server.go:199] starting sriov_dpu device plugin endpoint at: yusur.tech_sriov_dpu.sock I1108 14:43:33.474818 1 server.go:226] sriov_dpu device plugin endpoint started serving I1108 14:43:33.474842 1 main.go:77] All servers started. I1108 14:43:33.474848 1 main.go:78] Listening for term signals I1108 14:43:33.964954 1 server.go:110] Plugin: yusur.tech_sriov_dpu.sock gets registered successfully at Kubelet I1108 14:43:33.964962 1 server.go:134] ListAndWatch(sriov_dpu) invoked I1108 14:43:33.964969 1 server.go:142] ListAndWatch(sriov_dpu): send devices &ListAndWatchResponse{Devices:[]*Device{&Device{ID:0000:01:00.6,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.7,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.2,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.3,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.4,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.5,Health:Healthy,Topology:nil,},},}
Multus logs (If enabled. Try '/var/log/multus.log' )
Kubelet logs (journalctl -u kubelet)
I dont understand the issue,
device plugin is configured using rootDevices selector and is reporting 6 devices under ysur.tech/sriov_dpu
it emits warning in logs because it did not find netdevice for the PFs which is expected in your case since the PF driver did not create a netdevice.
CNI is failing since it failed to find netdevice.
I dont understand the issue,
device plugin is configured using
rootDevicesselector and is reporting 6 devices underysur.tech/sriov_dpuit emits warning in logs because it did not find netdevice for the PFs which is expected in your case since the PF driver did not create a netdevice.CNI is failing since it failed to find netdevice.
Sorry, let me rephrase.My current problem is that pod can’t find pfname
1. Situation
Now the directory of our virtio device on this machine is as follows. /sys/bus/pci/devices/0000:01:00.5/physfn/virtio5/net
The current version of sriov reads the pfname of this machine as shown below.
it such as /sys/bus/pci/devices/0000:01:00.5/physfn/net
It cannot find the net directory of this device, so it cannot be placed in the resource pool.
2. Solution
I added an extra logical judgment to the path here, and now it can be successfully adapted to our hardware equipment. We are a Chinese DPU equipment company YUSUR. Could you please evaluate whether it’s necessary to integrate this into the community?
PF name the the PF netdev name of the PCI device. what you suggest is to support in device plugin to take the name of the PF netdev is from a virtio device
saying in the topic the path is "wrong" sort of hints that device plugin behaves incorrectly, which is not the case.
I still dont understand what it is that you are seeking. currently device plugin will indeed fail to get PF name, but would still expose the resources.
if you want to make this virtio netdev available in workload pod then you need support in CNI not device plugin.
can you join one of our community meetings and present your use-case ? this would help
Certainly, I can attend the meeting. Meanwhile, please review my reproduction on your end. Please refrain from referring to my previous responses; focus on the replies I provide below.
1.Current original version:
#Local situation, where the node node1vdpa is a DPU node. I want to use our SR-IOV plugin to obtain the corresponding VF resources.
root@master:~/ovn/ovn-kubernetes/dist/images/autodeploy# kubectl get pods -A -owide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-bcb458c48-kxlbh 1/1 Running 0 7m10s 10.244.1.3 master <none> <none>
kube-system coredns-bcb458c48-qwqfn 1/1 Running 0 7m10s 10.244.1.4 master <none> <none>
kube-system etcd-master 1/1 Running 23 (60m ago) 45h 192.168.201.20 master <none> <none>
kube-system kube-apiserver-master 1/1 Running 23 (60m ago) 45h 192.168.201.20 master <none> <none>
kube-system kube-controller-manager-master 1/1 Running 23 (60m ago) 45h 192.168.201.20 master <none> <none>
kube-system kube-multus-ds-cd7nn 1/1 Running 0 7m9s 192.168.201.20 master <none> <none>
kube-system kube-multus-ds-nntrh 1/1 Running 0 7m9s 192.168.201.21 node1vdpa <none> <none>
kube-system kube-scheduler-master 1/1 Running 22 (60m ago) 45h 192.168.201.20 master <none> <none>
kube-system kube-sriov-device-plugin-amd64-5p72l 1/1 Running 0 27s 192.168.201.21 node1vdpa <none> <none>
kube-system kube-sriov-device-plugin-amd64-vgd9j 1/1 Running 0 7m9s 192.168.201.20 master <none> <none>
ovn-kubernetes ovnkube-db-77b8bbd66f-ccwpk 2/2 Running 0 11m 192.168.201.20 master <none> <none>
ovn-kubernetes ovnkube-master-78b7865c59-xntjb 2/2 Running 0 10m 192.168.201.20 master <none> <none>
ovn-kubernetes ovnkube-node-4vs7s 3/3 Running 0 9m12s 192.168.201.20 master <none> <none>
ovn-kubernetes ovnkube-node-dpu-host-vdpa-cv7m2 1/1 Running 0 8m11s 192.168.201.21 node1vdpa <none> <none>
ovn-kubernetes ovs-node-2w6qr 1/1 Running 0 12m 192.168.201.20 master <none> <none>
Let's first take a look at the log information for the SR-IOV pod.
#SR-IOV log information
root@master:~/ovn/ovn-kubernetes/dist/images/autodeploy# kubectl logs -f -n kube-system kube-sriov-device-plugin-amd64-5p72l
I1123 02:59:05.191954 1 manager.go:51] Using Kubelet Plugin Registry Mode
I1123 02:59:05.192234 1 main.go:44] resource manager reading configs
I1123 02:59:05.192262 1 manager.go:79] raw ResourceList: {
"resourceList": [{
"resourceName": "sriov_dpu",
"resourcePrefix": "yusur.tech",
"selectors": {
"pfNames": ["enp1s0#1-6"]
}
}
]
}
I1123 02:59:05.192348 1 factory.go:166] net device selector for resource sriov_dpu is &{DeviceSelectors:{Vendors:[] Devices:[] Drivers:[] PciAddresses:[]} PfNames:[enp1s0#1-6] RootDevices:[] LinkTypes:[] DDPProfiles:[] IsRdma:false NeedVhostNet:false}
I1123 02:59:05.192370 1 manager.go:99] unmarshalled ResourceList: [{ResourcePrefix:yusur.tech ResourceName:sriov_dpu DeviceType:netDevice Selectors:0xc00000cd38 SelectorObj:0xc000243110}]
I1123 02:59:05.192408 1 manager.go:200] validating resource name "yusur.tech/sriov_dpu"
I1123 02:59:05.192412 1 main.go:60] Discovering host devices
I1123 02:59:05.269156 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.0 02 unknown unknown
I1123 02:59:05.275501 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.1 02 unknown unknown
I1123 02:59:05.275589 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.2 02 unknown unknown
I1123 02:59:05.275653 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.3 02 unknown unknown
I1123 02:59:05.275702 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.4 02 unknown unknown
I1123 02:59:05.275752 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.5 02 unknown unknown
I1123 02:59:05.275799 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.6 02 unknown unknown
I1123 02:59:05.275864 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.7 02 unknown unknown
I1123 02:59:05.275920 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0b.0 02 unknown unknown
I1123 02:59:05.275984 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0c.0 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276007 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.0 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276029 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.1 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276042 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.2 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276052 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.3 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276061 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.4 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276072 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.5 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276082 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.6 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276093 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.7 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276210 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:02:00.0 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276239 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:10:00.0 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276291 1 main.go:66] Initializing resource servers
I1123 02:59:05.276435 1 manager.go:105] number of config: 1
I1123 02:59:05.276441 1 manager.go:109]
I1123 02:59:05.276444 1 manager.go:110] Creating new ResourcePool: sriov_dpu
I1123 02:59:05.276446 1 manager.go:111] DeviceType: netDevice
I1123 02:59:05.276764 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.1. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1123 02:59:05.276982 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.2. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1123 02:59:05.277159 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.3. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1123 02:59:05.277376 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.4. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1123 02:59:05.277596 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.5. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1123 02:59:05.277821 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.6. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1123 02:59:05.278080 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.7. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1123 02:59:05.278169 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0b.0. error getting devlink device attributes for net device 0000:00:0b.0 no such device
I1123 02:59:05.278261 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0c.0. error getting devlink device attributes for net device 0000:00:0c.0 no such device
W1123 02:59:05.278273 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:00:0c.0/net: no such file or directory"
I1123 02:59:05.278479 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.1. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1123 02:59:05.278489 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.1/net: no such file or directory"
I1123 02:59:05.278618 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.2. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1123 02:59:05.278627 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.2/net: no such file or directory"
I1123 02:59:05.278768 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.3. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1123 02:59:05.278777 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.3/net: no such file or directory"
I1123 02:59:05.278941 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.4. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1123 02:59:05.278952 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.4/net: no such file or directory"
I1123 02:59:05.279140 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.5. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1123 02:59:05.279149 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.5/net: no such file or directory"
I1123 02:59:05.279351 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.6. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1123 02:59:05.279359 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.6/net: no such file or directory"
I1123 02:59:05.279603 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.7. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1123 02:59:05.279612 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.7/net: no such file or directory"
I1123 02:59:05.279670 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:02:00.0. error getting devlink device attributes for net device 0000:02:00.0 no such device
W1123 02:59:05.279679 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:02:00.0/net: no such file or directory"
I1123 02:59:05.279738 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:10:00.0. error getting devlink device attributes for net device 0000:10:00.0 no such device
W1123 02:59:05.279747 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:10:00.0/net: no such file or directory"
I1123 02:59:05.279755 1 manager.go:125] no devices in device pool, skipping creating resource server for sriov_dpu
I1123 02:59:05.279758 1 main.go:72] Starting all servers...
I1123 02:59:05.279760 1 main.go:77] All servers started.
I1123 02:59:05.279763 1 main.go:78] Listening for term signals
it will throw error "unable to get PF name "open /sys/bus/pci/devices/0000:02:00.0/net: no such file or directory" Below is the node resource information. We can see that the node has not obtained SR-IOV resource information: yusur.tech/sriov_dpu: 0
#Node Resource Information
root@master:~/ovn/ovn-kubernetes/dist/images/autodeploy# kubectl describe node node1vdpa
Name: node1vdpa
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
k8s.ovn.org/dpu-host=
k8s.ovn.org/vdpa-dpu-host=
kubernetes.io/arch=amd64
kubernetes.io/hostname=node1vdpa
kubernetes.io/os=linux
node=node1
yusur-type=dpu
Annotations: k8s.ovn.org/host-addresses: ["172.17.0.1","192.168.124.1","192.168.200.35","192.168.201.101"]
k8s.ovn.org/l3-gateway-config:
{"default":{"mode":"shared","interface-id":"brp0_node1vdpa","mac-address":"00:15:6d:ee:ce:cb","ip-addresses":["192.168.201.21/24"],"ip-add...
k8s.ovn.org/node-chassis-id: 58674eff-bed3-4cec-837d-afc88df23dc4
k8s.ovn.org/node-gateway-router-lrp-ifaddr: {"ipv4":"100.64.0.3/16"}
k8s.ovn.org/node-mgmt-port-mac-address: 0a:58:0a:f4:00:02
k8s.ovn.org/node-primary-ifaddr: {"ipv4":"192.168.201.21/24"}
k8s.ovn.org/node-subnets: {"default":"10.244.0.0/24"}
kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 21 Nov 2023 13:51:38 +0800
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: node1vdpa
AcquireTime: <unset>
RenewTime: Thu, 23 Nov 2023 11:04:08 +0800
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Thu, 23 Nov 2023 10:59:16 +0800 Wed, 22 Nov 2023 19:44:57 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 23 Nov 2023 10:59:16 +0800 Wed, 22 Nov 2023 19:44:57 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 23 Nov 2023 10:59:16 +0800 Wed, 22 Nov 2023 19:44:57 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 23 Nov 2023 10:59:16 +0800 Thu, 23 Nov 2023 10:51:27 +0800 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 192.168.201.21
Hostname: node1vdpa
Capacity:
cpu: 8
ephemeral-storage: 24590672Ki
hugepages-1Gi: 0
hugepages-2Mi: 1200Mi
memory: 8127040Ki
pods: 110
yusur.tech/sriov_dpu: 6
Allocatable:
cpu: 8
ephemeral-storage: 22662763278
hugepages-1Gi: 0
hugepages-2Mi: 1200Mi
memory: 6795840Ki
pods: 110
yusur.tech/sriov_dpu: 0
System Info:
Machine ID: eabba479306945cfb24292224ef23d1a
System UUID: da7512b0-6152-49be-9bf6-de13d986c2c7
Boot ID: a2a0a5dc-308e-4db6-a362-8c3d7a855a3a
Kernel Version: 5.15.0-87-generic
OS Image: Ubuntu 20.04.6 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.6.19
Kubelet Version: v1.24.2
Kube-Proxy Version: v1.24.2
PodCIDR: 10.244.1.0/24
PodCIDRs: 10.244.1.0/24
Non-terminated Pods: (3 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system kube-multus-ds-nntrh 100m (1%) 100m (1%) 50Mi (0%) 50Mi (0%) 11m
kube-system kube-sriov-device-plugin-amd64-5p72l 250m (3%) 1 (12%) 40Mi (0%) 200Mi (3%) 5m7s
ovn-kubernetes ovnkube-node-dpu-host-vdpa-cv7m2 100m (1%) 0 (0%) 300Mi (4%) 0 (0%) 12m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 450m (5%) 1100m (13%)
memory 390Mi (5%) 250Mi (3%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 300Mi (25%) 300Mi (25%)
yusur.tech/sriov_dpu 0 0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NodeNotReady 27m (x2 over 15h) kubelet Node node1vdpa status is now: NodeNotReady
Normal NodeReady 12m (x4 over 45h) kubelet Node node1vdpa status is now: NodeReady
In other words, before modifying the code version, our SR-IOV cannot obtain node information in the current situation when using selectors for pfNames. At the same time, it will throw an error, "unable to get PF name "open /sys/bus/pci/devices/0000:01:00.3/net: no such file or directory". To understand the error situation, let's go to the node node1vdpa and investigate what's going on.
2. Node information for node1vdpa
Let's first take a look at the top-level directory. Currently, we are on the node1vdpa node.
root@node1vdpa:/sys/bus/pci/devices# ls
0000:00:00.0 0000:00:01.2 0000:00:01.5 0000:00:02.0 0000:00:02.3 0000:00:02.6 0000:00:0a.0 0000:00:0a.3 0000:00:0a.6 0000:00:0c.0 0000:00:1f.2 0000:01:00.1 0000:01:00.4 0000:01:00.7 0000:0b:00.0 0000:0e:00.0
0000:00:01.0 0000:00:01.3 0000:00:01.6 0000:00:02.1 0000:00:02.4 0000:00:02.7 0000:00:0a.1 0000:00:0a.4 0000:00:0a.7 0000:00:1b.0 0000:00:1f.3 0000:01:00.2 0000:01:00.5 0000:02:00.0 0000:0c:00.0 0000:10:00.0
0000:00:01.1 0000:00:01.4 0000:00:01.7 0000:00:02.2 0000:00:02.5 0000:00:03.0 0000:00:0a.2 0000:00:0a.5 0000:00:0b.0 0000:00:1f.0 0000:01:00.0 0000:01:00.3 0000:01:00.6 0000:09:00.0 0000:0d:00.0
root@node1vdpa:/sys/bus/pci/devices# cd /sys/bus/pci/devices/0000:01:00.3/
root@node1vdpa:/sys/bus/pci/devices/0000:01:00.3# ls
ari_enabled config current_link_width dma_mask_bits enable irq local_cpus modalias numa_node power_state resource resource4_wc subsystem uevent
broken_parity_status consistent_dma_mask_bits d3cold_allowed driver iommu link max_link_speed msi_bus physfn reset resource1 revision subsystem_device vendor
class current_link_speed device driver_override iommu_group local_cpulist max_link_width msi_irqs power reset_method resource4 sriov_vf_msix_count subsystem_vendor virtio10
After entering the directory /sys/bus/pci/devices/0000:01:00.3, we can indeed see that the net folder is not present. Let's take a look at how the code is implemented in the project to understand this.
We can see that the current code navigates into the physfn directory within /sys/bus/pci/devices/0000:01:00.3 to look for the net folder.
#let's see physfn dir
root@node1vdpa:/sys/bus/pci/devices/0000:01:00.3# cd physfn
# we cant find the net dir meanwhile we can find virtio1
root@node1vdpa:/sys/bus/pci/devices/0000:01:00.3/physfn# ls
ari_enabled consistent_dma_mask_bits device enable irq max_link_speed msi_irqs remove resource revision sriov_offset sriov_vf_total_msix uevent virtfn2 virtfn6
broken_parity_status current_link_speed dma_mask_bits firmware_node link max_link_width numa_node rescan resource1 rom sriov_stride subsystem vendor virtfn3 virtio1
class current_link_width driver iommu local_cpulist modalias power reset resource4 sriov_drivers_autoprobe sriov_totalvfs subsystem_device virtfn0 virtfn4
config d3cold_allowed driver_override iommu_group local_cpus msi_bus power_state reset_method resource4_wc sriov_numvfs sriov_vf_device subsystem_vendor virtfn1 virtfn5
It's evident that the net folder is not present in the current directory, so it cannot find the corresponding resources. As a result, the local SR-IOV resources are reported as 0.
root@node1vdpa:/sys/bus/pci/devices/0000:01:00.3/physfn# ethtool -i enp1s0
driver: virtio_net
version: 1.0.0
firmware-version:
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
3. Solution
When the driver is virtio, we can observe that, in the directory /sys/bus/pci/devices/0000:01:00.3/physfn, an additional folder named virtioxx is present. Let's enter that folder.
root@node1vdpa:/sys/bus/pci/devices/0000:01:00.3/physfn# ls
ari_enabled consistent_dma_mask_bits device enable irq max_link_speed msi_irqs remove resource revision sriov_offset sriov_vf_total_msix uevent virtfn2 virtfn6
broken_parity_status current_link_speed dma_mask_bits firmware_node link max_link_width numa_node rescan resource1 rom sriov_stride subsystem vendor virtfn3 virtio1
class current_link_width driver iommu local_cpulist modalias power reset resource4 sriov_drivers_autoprobe sriov_totalvfs subsystem_device virtfn0 virtfn4
config d3cold_allowed driver_override iommu_group local_cpus msi_bus power_state reset_method resource4_wc sriov_numvfs sriov_vf_device subsystem_vendor virtfn1 virtfn5
# virtio1 is our focus
root@node1vdpa:/sys/bus/pci/devices/0000:01:00.3/physfn# cd virtio1/
root@node1vdpa:/sys/bus/pci/devices/0000:01:00.3/physfn/virtio1# ls
device driver features modalias net power status subsystem uevent vendor
root@node1vdpa:/sys/bus/pci/devices/0000:01:00.6/physfn/virtio1# ls net/
enp1s0
Great, we can now see the net folder with enp1s0 inside, which is the pfName we need. So, the code can be modified as follows:
4. The effect after the modification.
#I have already switched to the image with the modified code on node1vdpa. Now, let's delete the existing pod
root@master:~/ovn/ovn-kubernetes/dist/images/autodeploy# kubectl delete po -n kube-system kube-sriov-device-plugin-amd64-5p72l --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "kube-sriov-device-plugin-amd64-5p72l" force deleted
#get pods
root@master:~/ovn/ovn-kubernetes/dist/images/autodeploy# kubectl get pods -A -owide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-bcb458c48-kxlbh 1/1 Running 0 45m 10.244.1.3 master <none> <none>
kube-system coredns-bcb458c48-qwqfn 1/1 Running 0 45m 10.244.1.4 master <none> <none>
kube-system etcd-master 1/1 Running 23 (98m ago) 45h 192.168.201.20 master <none> <none>
kube-system kube-apiserver-master 1/1 Running 23 (98m ago) 45h 192.168.201.20 master <none> <none>
kube-system kube-controller-manager-master 1/1 Running 23 (98m ago) 45h 192.168.201.20 master <none> <none>
kube-system kube-multus-ds-cd7nn 1/1 Running 0 45m 192.168.201.20 master <none> <none>
kube-system kube-multus-ds-nntrh 1/1 Running 0 45m 192.168.201.21 node1vdpa <none> <none>
kube-system kube-scheduler-master 1/1 Running 22 (98m ago) 45h 192.168.201.20 master <none> <none>
kube-system kube-sriov-device-plugin-amd64-8g52f 1/1 Running 0 3s 192.168.201.21 node1vdpa <none> <none>
kube-system kube-sriov-device-plugin-amd64-vgd9j 1/1 Running 0 45m 192.168.201.20 master <none> <none>
ovn-kubernetes ovnkube-db-77b8bbd66f-ccwpk 2/2 Running 0 49m 192.168.201.20 master <none> <none>
ovn-kubernetes ovnkube-master-78b7865c59-xntjb 2/2 Running 0 48m 192.168.201.20 master <none> <none>
ovn-kubernetes ovnkube-node-4vs7s 3/3 Running 0 47m 192.168.201.20 master <none> <none>
ovn-kubernetes ovnkube-node-dpu-host-vdpa-cv7m2 1/1 Running 0 46m 192.168.201.21 node1vdpa <none> <none>
ovn-kubernetes ovs-node-2w6qr 1/1 Running 0 50m 192.168.201.20 master <none> <none>
#check the pod logs.
root@master:~/ovn/ovn-kubernetes/dist/images/autodeploy# kubectl logs -f -n kube-system kube-sriov-device-plugin-amd64-8g52f
I1123 03:37:46.852262 1 manager.go:57] Using Kubelet Plugin Registry Mode
I1123 03:37:46.852408 1 main.go:46] resource manager reading configs
I1123 03:37:46.852435 1 manager.go:86] raw ResourceList: {
"resourceList": [{
"resourceName": "sriov_dpu",
"resourcePrefix": "yusur.tech",
"selectors": {
"pfNames": ["enp1s0#1-6"]
}
}
]
}
I1123 03:37:46.852517 1 factory.go:209] *types.NetDeviceSelectors for resource sriov_dpu is [0xc0002a0e00]
I1123 03:37:46.852526 1 manager.go:106] unmarshalled ResourceList: [{ResourcePrefix:yusur.tech ResourceName:sriov_dpu DeviceType:netDevice ExcludeTopology:false Selectors:0xc00041a840 AdditionalInfo:map[] SelectorObjs:[0xc0002a0e00]}]
I1123 03:37:46.852554 1 manager.go:217] validating resource name "yusur.tech/sriov_dpu"
I1123 03:37:46.852562 1 main.go:62] Discovering host devices
I1123 03:37:46.891928 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.0 02 unknown unknown
I1123 03:37:46.892463 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.1 02 unknown unknown
I1123 03:37:46.892618 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.2 02 unknown unknown
I1123 03:37:46.892702 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.3 02 unknown unknown
I1123 03:37:46.892775 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.4 02 unknown unknown
I1123 03:37:46.892840 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.5 02 unknown unknown
I1123 03:37:46.892903 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.6 02 unknown unknown
I1123 03:37:46.892966 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.7 02 unknown unknown
I1123 03:37:46.893028 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0b.0 02 unknown unknown
I1123 03:37:46.893111 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0c.0 02 Red Hat, Inc. Virtio network device
I1123 03:37:46.893135 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.0 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893164 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.1 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893176 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.2 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893186 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.3 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893197 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.4 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893208 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.5 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893221 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.6 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893234 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.7 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893248 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:02:00.0 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893276 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:10:00.0 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893304 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.0 02 unknown unknown
I1123 03:37:46.893321 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.1 02 unknown unknown
I1123 03:37:46.893324 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.2 02 unknown unknown
I1123 03:37:46.893327 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.3 02 unknown unknown
I1123 03:37:46.893330 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.4 02 unknown unknown
I1123 03:37:46.893335 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.5 02 unknown unknown
I1123 03:37:46.893338 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.6 02 unknown unknown
I1123 03:37:46.893341 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.7 02 unknown unknown
I1123 03:37:46.893346 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0b.0 02 unknown unknown
I1123 03:37:46.893349 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0c.0 02 Red Hat, Inc. Virtio network device
I1123 03:37:46.893352 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.0 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893357 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.1 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893363 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.2 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893367 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.3 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893371 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.4 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893373 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.5 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893378 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.6 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893381 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.7 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893384 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:02:00.0 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893392 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:10:00.0 02 Red Hat, Inc. Virtio 1.0 network device
I1123 03:37:46.893395 1 main.go:68] Initializing resource servers
I1123 03:37:46.893402 1 manager.go:117] number of config: 1
I1123 03:37:46.893407 1 manager.go:121] Creating new ResourcePool: sriov_dpu
I1123 03:37:46.893410 1 manager.go:122] DeviceType: netDevice
I1123 03:37:46.893505 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.1. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1123 03:37:46.893752 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.2. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1123 03:37:46.893982 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.3. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1123 03:37:46.894218 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.4. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1123 03:37:46.894488 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.5. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1123 03:37:46.894771 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.6. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1123 03:37:46.895066 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.7. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1123 03:37:46.895460 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.1. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1123 03:37:46.895659 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.2. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1123 03:37:46.895879 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.3. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1123 03:37:46.896124 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.4. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1123 03:37:46.896395 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.5. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1123 03:37:46.896670 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.6. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1123 03:37:46.896956 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.7. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1123 03:37:46.897259 1 manager.go:138] initServers(): selector index 0 will register 6 devices
I1123 03:37:46.897269 1 factory.go:122] device added: [identifier: 0000:01:00.2, vendor: 1af4, device: 1041, driver: virtio-pci]
I1123 03:37:46.897272 1 factory.go:122] device added: [identifier: 0000:01:00.3, vendor: 1af4, device: 1041, driver: virtio-pci]
I1123 03:37:46.897275 1 factory.go:122] device added: [identifier: 0000:01:00.4, vendor: 1af4, device: 1041, driver: virtio-pci]
I1123 03:37:46.897277 1 factory.go:122] device added: [identifier: 0000:01:00.5, vendor: 1af4, device: 1041, driver: virtio-pci]
I1123 03:37:46.897280 1 factory.go:122] device added: [identifier: 0000:01:00.6, vendor: 1af4, device: 1041, driver: vfio-pci]
I1123 03:37:46.897284 1 factory.go:122] device added: [identifier: 0000:01:00.7, vendor: 1af4, device: 1041, driver: virtio-pci]
I1123 03:37:46.897303 1 manager.go:156] New resource server is created for sriov_dpu ResourcePool
I1123 03:37:46.897309 1 main.go:74] Starting all servers...
I1123 03:37:46.897475 1 server.go:254] starting sriov_dpu device plugin endpoint at: yusur.tech_sriov_dpu.sock
I1123 03:37:46.898073 1 server.go:282] sriov_dpu device plugin endpoint started serving
I1123 03:37:46.898109 1 main.go:79] All servers started.
I1123 03:37:46.898115 1 main.go:80] Listening for term signals
I1123 03:37:47.847008 1 server.go:116] Plugin: yusur.tech_sriov_dpu.sock gets registered successfully at Kubelet
I1123 03:37:47.847085 1 server.go:157] ListAndWatch(sriov_dpu) invoked
I1123 03:37:47.847107 1 server.go:170] ListAndWatch(sriov_dpu): send devices &ListAndWatchResponse{Devices:[]*Device{&Device{ID:0000:01:00.7,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.2,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.3,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.4,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.5,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.6,Health:Healthy,Topology:nil,},},}
Let's check the SR-IOV resources on the local node.
root@master:~/ovn/ovn-kubernetes/dist/images/autodeploy# kubectl describe node node1vdpa
Name: node1vdpa
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
k8s.ovn.org/dpu-host=
k8s.ovn.org/vdpa-dpu-host=
kubernetes.io/arch=amd64
kubernetes.io/hostname=node1vdpa
kubernetes.io/os=linux
node=node1
yusur-type=dpu
Annotations: k8s.ovn.org/host-addresses: ["172.17.0.1","192.168.124.1","192.168.200.35","192.168.201.101"]
k8s.ovn.org/l3-gateway-config:
{"default":{"mode":"shared","interface-id":"brp0_node1vdpa","mac-address":"00:15:6d:ee:ce:cb","ip-addresses":["192.168.201.21/24"],"ip-add...
k8s.ovn.org/node-chassis-id: 58674eff-bed3-4cec-837d-afc88df23dc4
k8s.ovn.org/node-gateway-router-lrp-ifaddr: {"ipv4":"100.64.0.3/16"}
k8s.ovn.org/node-mgmt-port-mac-address: 0a:58:0a:f4:00:02
k8s.ovn.org/node-primary-ifaddr: {"ipv4":"192.168.201.21/24"}
k8s.ovn.org/node-subnets: {"default":"10.244.0.0/24"}
kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 21 Nov 2023 13:51:38 +0800
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: node1vdpa
AcquireTime: <unset>
RenewTime: Thu, 23 Nov 2023 11:39:37 +0800
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Thu, 23 Nov 2023 11:37:55 +0800 Wed, 22 Nov 2023 19:44:57 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 23 Nov 2023 11:37:55 +0800 Wed, 22 Nov 2023 19:44:57 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 23 Nov 2023 11:37:55 +0800 Wed, 22 Nov 2023 19:44:57 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 23 Nov 2023 11:37:55 +0800 Thu, 23 Nov 2023 10:51:27 +0800 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 192.168.201.21
Hostname: node1vdpa
Capacity:
cpu: 8
ephemeral-storage: 24590672Ki
hugepages-1Gi: 0
hugepages-2Mi: 1200Mi
memory: 8127040Ki
pods: 110
yusur.tech/sriov_dpu: 6
Allocatable:
cpu: 8
ephemeral-storage: 22662763278
hugepages-1Gi: 0
hugepages-2Mi: 1200Mi
memory: 6795840Ki
pods: 110
yusur.tech/sriov_dpu: 6
System Info:
Machine ID: eabba479306945cfb24292224ef23d1a
System UUID: da7512b0-6152-49be-9bf6-de13d986c2c7
Boot ID: a2a0a5dc-308e-4db6-a362-8c3d7a855a3a
Kernel Version: 5.15.0-87-generic
OS Image: Ubuntu 20.04.6 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.6.19
Kubelet Version: v1.24.2
Kube-Proxy Version: v1.24.2
PodCIDR: 10.244.1.0/24
PodCIDRs: 10.244.1.0/24
Non-terminated Pods: (3 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system kube-multus-ds-nntrh 100m (1%) 100m (1%) 50Mi (0%) 50Mi (0%) 47m
kube-system kube-sriov-device-plugin-amd64-8g52f 250m (3%) 1 (12%) 40Mi (0%) 200Mi (3%) 2m
ovn-kubernetes ovnkube-node-dpu-host-vdpa-cv7m2 100m (1%) 0 (0%) 300Mi (4%) 0 (0%) 48m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 450m (5%) 1100m (13%)
memory 390Mi (5%) 250Mi (3%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 300Mi (25%) 300Mi (25%)
yusur.tech/sriov_dpu 0 0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NodeReady 48m (x4 over 45h) kubelet Node node1vdpa status is now: NodeReady
Hi, please be aware that pfName is not for regular interfaces. you need to use the pciAddress if you want to add the devices to the sriov-device-plugin.
if not you will have an issue if you have a pod running using the device and you restart the device plugin he will not be able to find the device after the reboot.
@JasinlikeEatingOrange can you change pfName selector to rootDevices selector and use PCI address of the physical device ? this selector also support VF index range syntax
@SchSeba pciAddress selector will match on the exact device, i believe he needs rootDevices to get all VF devices of a given physical device.
but they are not VFs if I didn't miss anything from the logs you can see
I1123 02:59:05.269156 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.0 02 unknown unknown
I1123 02:59:05.275501 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.1 02 unknown unknown
I1123 02:59:05.275589 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.2 02 unknown unknown
I1123 02:59:05.275653 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.3 02 unknown unknown
I1123 02:59:05.275702 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.4 02 unknown unknown
I1123 02:59:05.275752 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.5 02 unknown unknown
I1123 02:59:05.275799 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.6 02 unknown unknown
I1123 02:59:05.275864 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.7 02 unknown unknown
I1123 02:59:05.275920 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0b.0 02 unknown unknown
I1123 02:59:05.275984 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0c.0 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276007 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.0 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276029 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.1 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276042 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.2 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276052 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.3 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276061 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.4 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276072 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.5 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276082 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.6 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276093 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.7 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276210 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:02:00.0 02 Red Hat, Inc. Virtio network device
I1123 02:59:05.276239 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:10:00.0 02 Red Hat, Inc. Virtio network device
they are all Red Hat, Inc. Virtio network device these is not a PF they are just virtual devices.
I can also see Capacity: cpu: 8 so I am pretty sure it's a virtual server.
these are the only selectors supported for devices that are not sriov PFs
"pciAddresses" N Target device's pci address as string string list Default: null "pciAddresses": ["0000:03:02.0"] "acpiIndexes" N Target device's acpi index as string string list Default: null "acpiIndexes": ["101"]
but they are not VFs if I didn't miss anything from the logs you can see
from previous comment https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin/issues/507#issuecomment-1823825669
see section 2.
the pci device has a physfn link in sysfs and the physical device has virtfn files in sysfs, so was assuming its sriov. maybe its not the case IDK...
I don't understand also.
@JasinlikeEatingOrange can you please explain what is this device as it doesn't look like a real PF
I don't understand also.
@JasinlikeEatingOrange can you please explain what is this device as it doesn't look like a real PF
@SchSeba @adrianchiris I apologize for not explaining this earlier. As a DPU manufacturer, Our actual DPU hardware devices are relatively expensive, with each DPU physical card costing approximately tens of thousands of dollars. Therefore, for development and testing purposes, we use several virtual machines to simulate these devices. When it comes to final online validation, we use physical cards. However, it's important to note that the code for simulating PF (Physical Function) and VF (Virtual Function) in the virtual environment is identical to that of the real hardware devices.what we are currently observing can be understood as a genuine physical network card with multiple virtual functions being created. I answered your previous question. Does my response align with the inquiry from before? I'm concerned that I may have misunderstood it.
does rootDevices selector OR pciAddresses work for your virtual env ?
what DPU are we talking about ? company model etc ?
@JasinlikeEatingOrange base on
However, it's important to note that the code for simulating PF (Physical Function) and VF (Virtual Function) in the virtual environment is identical to that of the real hardware devices.what we are currently observing can be understood as a genuine physical network card with multiple virtual functions being created.
Does this mean that the PF for your card is going to be a virtio device?
rootDevices it works but it will throw error W1208 12:24:37.819646 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.5/net: no such file or directory"
I1208 12:24:37.632987 1 manager.go:51] Using Kubelet Plugin Registry Mode
I1208 12:24:37.647684 1 main.go:44] resource manager reading configs
I1208 12:24:37.647717 1 manager.go:79] raw ResourceList: {
"resourceList": [{
"resourceName": "sriov_dpu",
"resourcePrefix": "yusur.tech",
"selectors": {
"rootDevices": ["0000:01:00.0#1-6"]
}
}
]
}
I1208 12:24:37.647806 1 factory.go:166] net device selector for resource sriov_dpu is &{DeviceSelectors:{Vendors:[] Devices:[] Drivers:[] PciAddresses:[]} PfNames:[] RootDevices:[0000:01:00.0#1-6] LinkTypes:[] DDPProfiles:[] IsRdma:false NeedVhostNet:false}
I1208 12:24:37.647831 1 manager.go:99] unmarshalled ResourceList: [{ResourcePrefix:yusur.tech ResourceName:sriov_dpu DeviceType:netDevice Selectors:0xc00000cd38 SelectorObj:0xc0001c3110}]
I1208 12:24:37.647874 1 manager.go:200] validating resource name "yusur.tech/sriov_dpu"
I1208 12:24:37.647879 1 main.go:60] Discovering host devices
I1208 12:24:37.810009 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.0 02 unknown unknown
I1208 12:24:37.810358 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.1 02 unknown unknown
I1208 12:24:37.810450 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.2 02 unknown unknown
I1208 12:24:37.816184 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.3 02 unknown unknown
I1208 12:24:37.816267 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.4 02 unknown unknown
I1208 12:24:37.816325 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.5 02 unknown unknown
I1208 12:24:37.816377 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.6 02 unknown unknown
I1208 12:24:37.816440 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.7 02 unknown unknown
I1208 12:24:37.816500 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0b.0 02 unknown unknown
I1208 12:24:37.816576 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0c.0 02 Red Hat, Inc. Virtio network device
I1208 12:24:37.816594 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.0 02 Red Hat, Inc. Virtio network device
I1208 12:24:37.816615 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.1 02 Red Hat, Inc. Virtio network device
I1208 12:24:37.816625 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.2 02 Red Hat, Inc. Virtio network device
I1208 12:24:37.816636 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.3 02 Red Hat, Inc. Virtio network device
I1208 12:24:37.816647 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.4 02 Red Hat, Inc. Virtio network device
I1208 12:24:37.816657 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.5 02 Red Hat, Inc. Virtio network device
I1208 12:24:37.816667 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.6 02 Red Hat, Inc. Virtio network device
I1208 12:24:37.816679 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.7 02 Red Hat, Inc. Virtio network device
I1208 12:24:37.816689 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:02:00.0 02 Red Hat, Inc. Virtio network device
I1208 12:24:37.816717 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:10:00.0 02 Red Hat, Inc. Virtio network device
I1208 12:24:37.816740 1 main.go:66] Initializing resource servers
I1208 12:24:37.816748 1 manager.go:105] number of config: 1
I1208 12:24:37.816751 1 manager.go:109]
I1208 12:24:37.816753 1 manager.go:110] Creating new ResourcePool: sriov_dpu
I1208 12:24:37.816756 1 manager.go:111] DeviceType: netDevice
I1208 12:24:37.816968 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.1. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1208 12:24:37.817177 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.2. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1208 12:24:37.817377 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.3. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1208 12:24:37.817597 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.4. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1208 12:24:37.817839 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.5. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1208 12:24:37.818106 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.6. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1208 12:24:37.818391 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.7. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1208 12:24:37.818496 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0b.0. error getting devlink device attributes for net device 0000:00:0b.0 no such device
I1208 12:24:37.818586 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0c.0. error getting devlink device attributes for net device 0000:00:0c.0 no such device
W1208 12:24:37.818595 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:00:0c.0/net: no such file or directory"
I1208 12:24:37.818833 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.1. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1208 12:24:37.818844 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.1/net: no such file or directory"
I1208 12:24:37.818989 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.2. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1208 12:24:37.818997 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.2/net: no such file or directory"
I1208 12:24:37.819162 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.3. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1208 12:24:37.819171 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.3/net: no such file or directory"
I1208 12:24:37.819360 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.4. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1208 12:24:37.819370 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.4/net: no such file or directory"
I1208 12:24:37.819637 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.5. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1208 12:24:37.819646 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.5/net: no such file or directory"
I1208 12:24:37.819888 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.6. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1208 12:24:37.819897 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.6/net: no such file or directory"
I1208 12:24:37.820160 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.7. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1208 12:24:37.820169 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.7/net: no such file or directory"
I1208 12:24:37.820232 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:02:00.0. error getting devlink device attributes for net device 0000:02:00.0 no such device
W1208 12:24:37.820242 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:02:00.0/net: no such file or directory"
I1208 12:24:37.820306 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:10:00.0. error getting devlink device attributes for net device 0000:10:00.0 no such device
W1208 12:24:37.820314 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:10:00.0/net: no such file or directory"
I1208 12:24:37.820330 1 factory.go:106] device added: [pciAddr: 0000:01:00.2, vendor: 1af4, device: 1041, driver: virtio-pci]
I1208 12:24:37.820333 1 factory.go:106] device added: [pciAddr: 0000:01:00.3, vendor: 1af4, device: 1041, driver: virtio-pci]
I1208 12:24:37.820336 1 factory.go:106] device added: [pciAddr: 0000:01:00.4, vendor: 1af4, device: 1041, driver: virtio-pci]
I1208 12:24:37.820338 1 factory.go:106] device added: [pciAddr: 0000:01:00.5, vendor: 1af4, device: 1041, driver: virtio-pci]
I1208 12:24:37.820340 1 factory.go:106] device added: [pciAddr: 0000:01:00.6, vendor: 1af4, device: 1041, driver: virtio-pci]
I1208 12:24:37.820343 1 factory.go:106] device added: [pciAddr: 0000:01:00.7, vendor: 1af4, device: 1041, driver: virtio-pci]
I1208 12:24:37.820352 1 manager.go:139] New resource server is created for sriov_dpu ResourcePool
I1208 12:24:37.820356 1 main.go:72] Starting all servers...
I1208 12:24:37.820537 1 server.go:199] starting sriov_dpu device plugin endpoint at: yusur.tech_sriov_dpu.sock
I1208 12:24:37.829701 1 server.go:226] sriov_dpu device plugin endpoint started serving
I1208 12:24:37.829722 1 main.go:77] All servers started.
I1208 12:24:37.829726 1 main.go:78] Listening for term signals
I1208 12:24:38.293372 1 server.go:110] Plugin: yusur.tech_sriov_dpu.sock gets registered successfully at Kubelet
I1208 12:24:38.293426 1 server.go:134] ListAndWatch(sriov_dpu) invoked
I1208 12:24:38.293431 1 server.go:142] ListAndWatch(sriov_dpu): send devices &ListAndWatchResponse{Devices:[]*Device{&Device{ID:0000:01:00.2,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.3,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.4,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.5,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.6,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.7,Health:Healthy,Topology:nil,},},}
pciAddresses Use pciAddresses to select vf resource.
root@master:~# kubectl logs -f -n kube-system kube-sriov-device-plugin-amd64-8rg72
I1208 12:31:14.488334 1 manager.go:51] Using Kubelet Plugin Registry Mode
I1208 12:31:14.488497 1 main.go:44] resource manager reading configs
I1208 12:31:14.488519 1 manager.go:79] raw ResourceList: {
"resourceList": [{
"resourceName": "sriov_dpu",
"resourcePrefix": "yusur.tech",
"selectors": {
"pciAddresses": ["0000:01:00.0#1-6"]
}
}
]
}
I1208 12:31:14.488632 1 factory.go:166] net device selector for resource sriov_dpu is &{DeviceSelectors:{Vendors:[] Devices:[] Drivers:[] PciAddresses:[0000:01:00.0#1-6]} PfNames:[] RootDevices:[] LinkTypes:[] DDPProfiles:[] IsRdma:false NeedVhostNet:false}
I1208 12:31:14.488645 1 manager.go:99] unmarshalled ResourceList: [{ResourcePrefix:yusur.tech ResourceName:sriov_dpu DeviceType:netDevice Selectors:0xc00011ad20 SelectorObj:0xc0001dd2b0}]
I1208 12:31:14.488678 1 manager.go:200] validating resource name "yusur.tech/sriov_dpu"
I1208 12:31:14.488682 1 main.go:60] Discovering host devices
I1208 12:31:14.631963 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.0 02 unknown unknown
I1208 12:31:14.632309 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.1 02 unknown unknown
I1208 12:31:14.632412 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.2 02 unknown unknown
I1208 12:31:14.632484 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.3 02 unknown unknown
I1208 12:31:14.632558 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.4 02 unknown unknown
I1208 12:31:14.632621 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.5 02 unknown unknown
I1208 12:31:14.632692 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.6 02 unknown unknown
I1208 12:31:14.632764 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0a.7 02 unknown unknown
I1208 12:31:14.632827 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0b.0 02 unknown unknown
I1208 12:31:14.632906 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:00:0c.0 02 Red Hat, Inc. Virtio network device
I1208 12:31:14.689364 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.0 02 Red Hat, Inc. Virtio network device
I1208 12:31:14.693363 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.1 02 Red Hat, Inc. Virtio network device
I1208 12:31:14.693388 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.2 02 Red Hat, Inc. Virtio network device
I1208 12:31:14.693399 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.3 02 Red Hat, Inc. Virtio network device
I1208 12:31:14.693408 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.4 02 Red Hat, Inc. Virtio network device
I1208 12:31:14.693652 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.5 02 Red Hat, Inc. Virtio network device
I1208 12:31:14.693673 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.6 02 Red Hat, Inc. Virtio network device
I1208 12:31:14.693687 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:01:00.7 02 Red Hat, Inc. Virtio network device
I1208 12:31:14.693700 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:02:00.0 02 Red Hat, Inc. Virtio network device
I1208 12:31:14.693928 1 netDeviceProvider.go:84] netdevice AddTargetDevices(): device found: 0000:10:00.0 02 Red Hat, Inc. Virtio network device
I1208 12:31:14.693966 1 main.go:66] Initializing resource servers
I1208 12:31:14.693974 1 manager.go:105] number of config: 1
I1208 12:31:14.693977 1 manager.go:109]
I1208 12:31:14.693980 1 manager.go:110] Creating new ResourcePool: sriov_dpu
I1208 12:31:14.693982 1 manager.go:111] DeviceType: netDevice
I1208 12:31:14.700548 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.1. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1208 12:31:14.700829 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.2. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1208 12:31:14.701085 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.3. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1208 12:31:14.701397 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.4. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1208 12:31:14.701727 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.5. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1208 12:31:14.702054 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.6. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1208 12:31:14.702368 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0a.7. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1208 12:31:14.705249 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0b.0. error getting devlink device attributes for net device 0000:00:0b.0 no such device
I1208 12:31:14.705367 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:00:0c.0. error getting devlink device attributes for net device 0000:00:0c.0 no such device
W1208 12:31:14.705382 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:00:0c.0/net: no such file or directory"
I1208 12:31:14.705721 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.1. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1208 12:31:14.705737 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.1/net: no such file or directory"
I1208 12:31:14.705913 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.2. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1208 12:31:14.705928 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.2/net: no such file or directory"
I1208 12:31:14.706125 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.3. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1208 12:31:14.706141 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.3/net: no such file or directory"
I1208 12:31:14.706353 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.4. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1208 12:31:14.706370 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.4/net: no such file or directory"
I1208 12:31:14.706613 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.5. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1208 12:31:14.706627 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.5/net: no such file or directory"
I1208 12:31:14.706932 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.6. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1208 12:31:14.706946 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.6/net: no such file or directory"
I1208 12:31:14.707229 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:01:00.7. error getting devlink device attributes for net device 0000:01:00.0 no such device
W1208 12:31:14.707242 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.7/net: no such file or directory"
I1208 12:31:14.707318 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:02:00.0. error getting devlink device attributes for net device 0000:02:00.0 no such device
W1208 12:31:14.707332 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:02:00.0/net: no such file or directory"
I1208 12:31:14.707410 1 utils.go:71] Devlink query for eswitch mode is not supported for device 0000:10:00.0. error getting devlink device attributes for net device 0000:10:00.0 no such device
W1208 12:31:14.707423 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:10:00.0/net: no such file or directory"
I1208 12:31:14.707431 1 manager.go:125] no devices in device pool, skipping creating resource server for sriov_dpu
I1208 12:31:14.707435 1 main.go:72] Starting all servers...
I1208 12:31:14.707438 1 main.go:77] All servers started.
I1208 12:31:14.707443 1 main.go:78] Listening for term signals
@JasinlikeEatingOrange base on
However, it's important to note that the code for simulating PF (Physical Function) and VF (Virtual Function) in the virtual environment is identical to that of the real hardware devices.what we are currently observing can be understood as a genuine physical network card with multiple virtual functions being created.
Does this mean that the PF for your card is going to be a virtio device?
Yes, Your understanding is correct. During the verification phase, this card is simulated by a virtual machine. When it's actually sold to customers, the displayed effects are consistent. So, you can simply understand the PF as a real physical device. There's no need to worry whether it's a real physical device or not. In fact, a real physical device would also have this error.
rootDevices it works but it will throw error W1208 12:24:37.819646 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.5/net: no such file or directory"
yes, its a warning [1] as expected. it would still expose related VFs as resources.
[1] https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin/blob/30652a258b292bd81b7efa6890461adf79654c52/pkg/devices/gen_net.go#L52****
rootDevices it works but it will throw error W1208 12:24:37.819646 1 pciNetDevice.go:81] unable to get PF name "open /sys/bus/pci/devices/0000:01:00.5/net: no such file or directory"
yes, its a warning [1] as expected. it would still expose related VFs as resources.
[1]
https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin/blob/30652a258b292bd81b7efa6890461adf79654c52/pkg/devices/gen_net.go#L52
yes if i use this modify image it will work, use pfname and rootDevices works. not throw warning. should this modify merge to repo?
[1]
sriov-network-device-plugin/pkg/devices/gen_net.go
contributor can determine if it's necessary to modify it.
root@master:~# kubectl logs -f -n kube-system kube-sriov-device-plugin-amd64-qgjzq
I1210 10:01:34.904510 1 manager.go:57] Using Kubelet Plugin Registry Mode
I1210 10:01:34.904700 1 main.go:46] resource manager reading configs
I1210 10:01:34.904730 1 manager.go:86] raw ResourceList: {
"resourceList": [{
"resourceName": "sriov_dpu",
"resourcePrefix": "yusur.tech",
"selectors": {
"rootDevices": ["0000:01:00.0#1-6"]
}
}
]
}
I1210 10:01:34.904827 1 factory.go:209] *types.NetDeviceSelectors for resource sriov_dpu is [0xc0001b8e00]
I1210 10:01:34.904839 1 manager.go:106] unmarshalled ResourceList: [{ResourcePrefix:yusur.tech ResourceName:sriov_dpu DeviceType:netDevice ExcludeTopology:false Selectors:0xc000332870 AdditionalInfo:map[] SelectorObjs:[0xc0001b8e00]}]
I1210 10:01:34.904868 1 manager.go:217] validating resource name "yusur.tech/sriov_dpu"
I1210 10:01:34.904875 1 main.go:62] Discovering host devices
I1210 10:01:35.040963 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.0 02 unknown unknown
I1210 10:01:35.041013 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.1 02 unknown unknown
I1210 10:01:35.041017 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.2 02 unknown unknown
I1210 10:01:35.041020 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.3 02 unknown unknown
I1210 10:01:35.041022 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.4 02 unknown unknown
I1210 10:01:35.041025 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.5 02 unknown unknown
I1210 10:01:35.041028 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.6 02 unknown unknown
I1210 10:01:35.041030 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.7 02 unknown unknown
I1210 10:01:35.041032 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0b.0 02 unknown unknown
I1210 10:01:35.041035 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0c.0 02 Red Hat, Inc. Virtio network device
I1210 10:01:35.041038 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.0 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.041040 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.1 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.041042 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.2 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.041044 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.3 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.041047 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.4 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.041049 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.5 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.041052 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.6 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.041057 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.7 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.041060 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:02:00.0 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.041063 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:10:00.0 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.041068 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.0 02 unknown unknown
I1210 10:01:35.041529 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.1 02 unknown unknown
I1210 10:01:35.041653 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.2 02 unknown unknown
I1210 10:01:35.041787 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.3 02 unknown unknown
I1210 10:01:35.041891 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.4 02 unknown unknown
I1210 10:01:35.041956 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.5 02 unknown unknown
I1210 10:01:35.042016 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.6 02 unknown unknown
I1210 10:01:35.042088 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.7 02 unknown unknown
I1210 10:01:35.042144 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0b.0 02 unknown unknown
I1210 10:01:35.042223 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0c.0 02 Red Hat, Inc. Virtio network device
I1210 10:01:35.042247 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.0 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.042271 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.1 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.042290 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.2 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.042300 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.3 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.042310 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.4 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.042322 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.5 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.042334 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.6 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.042346 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.7 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.042356 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:02:00.0 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.042379 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:10:00.0 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:01:35.042408 1 main.go:68] Initializing resource servers
I1210 10:01:35.042412 1 manager.go:117] number of config: 1
I1210 10:01:35.042416 1 manager.go:121] Creating new ResourcePool: sriov_dpu
I1210 10:01:35.042419 1 manager.go:122] DeviceType: netDevice
I1210 10:01:35.042498 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.1. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1210 10:01:35.042719 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.2. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1210 10:01:35.042938 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.3. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1210 10:01:35.043168 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.4. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1210 10:01:35.043421 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.5. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1210 10:01:35.043721 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.6. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1210 10:01:35.044029 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.7. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1210 10:01:35.044448 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.1. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1210 10:01:35.044640 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.2. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1210 10:01:35.044848 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.3. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1210 10:01:35.045078 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.4. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1210 10:01:35.045330 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.5. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1210 10:01:35.053897 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.6. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1210 10:01:35.054290 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.7. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1210 10:01:35.054732 1 manager.go:138] initServers(): selector index 0 will register 6 devices
I1210 10:01:35.054741 1 factory.go:122] device added: [identifier: 0000:01:00.2, vendor: 1af4, device: 1041, driver: virtio-pci]
I1210 10:01:35.054745 1 factory.go:122] device added: [identifier: 0000:01:00.3, vendor: 1af4, device: 1041, driver: virtio-pci]
I1210 10:01:35.054747 1 factory.go:122] device added: [identifier: 0000:01:00.4, vendor: 1af4, device: 1041, driver: virtio-pci]
I1210 10:01:35.054750 1 factory.go:122] device added: [identifier: 0000:01:00.5, vendor: 1af4, device: 1041, driver: virtio-pci]
I1210 10:01:35.054752 1 factory.go:122] device added: [identifier: 0000:01:00.6, vendor: 1af4, device: 1041, driver: virtio-pci]
I1210 10:01:35.054757 1 factory.go:122] device added: [identifier: 0000:01:00.7, vendor: 1af4, device: 1041, driver: virtio-pci]
I1210 10:01:35.054796 1 manager.go:156] New resource server is created for sriov_dpu ResourcePool
I1210 10:01:35.054803 1 main.go:74] Starting all servers...
I1210 10:01:35.055023 1 server.go:254] starting sriov_dpu device plugin endpoint at: yusur.tech_sriov_dpu.sock
I1210 10:01:35.057946 1 server.go:282] sriov_dpu device plugin endpoint started serving
I1210 10:01:35.057975 1 main.go:79] All servers started.
I1210 10:01:35.057981 1 main.go:80] Listening for term signals
I1210 10:01:35.714697 1 server.go:116] Plugin: yusur.tech_sriov_dpu.sock gets registered successfully at Kubelet
I1210 10:01:35.714778 1 server.go:157] ListAndWatch(sriov_dpu) invoked
I1210 10:01:35.714798 1 server.go:170] ListAndWatch(sriov_dpu): send devices &ListAndWatchResponse{Devices:[]*Device{&Device{ID:0000:01:00.2,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.3,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.4,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.5,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.6,Health:Healthy,Topology:nil,},&Device{ID:0000:01:00.7,Health:Healthy,Topology:nil,},},}
But use pciAddresses can't find resources.
root@master:~# kubectl logs -f -n kube-system kube-sriov-device-plugin-amd64-6w57p
I1210 10:10:19.221745 1 manager.go:57] Using Kubelet Plugin Registry Mode
I1210 10:10:19.237750 1 main.go:46] resource manager reading configs
I1210 10:10:19.237915 1 manager.go:86] raw ResourceList: {
"resourceList": [{
"resourceName": "sriov_dpu",
"resourcePrefix": "yusur.tech",
"selectors": {
"pciAddresses": ["0000:01:00.0#1-6"]
}
}
]
}
I1210 10:10:19.238234 1 factory.go:209] *types.NetDeviceSelectors for resource sriov_dpu is [0xc0002a0e00]
I1210 10:10:19.238247 1 manager.go:106] unmarshalled ResourceList: [{ResourcePrefix:yusur.tech ResourceName:sriov_dpu DeviceType:netDevice ExcludeTopology:false Selectors:0xc00041c840 AdditionalInfo:map[] SelectorObjs:[0xc0002a0e00]}]
I1210 10:10:19.238384 1 manager.go:217] validating resource name "yusur.tech/sriov_dpu"
I1210 10:10:19.238392 1 main.go:62] Discovering host devices
I1210 10:10:19.333593 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.0 02 unknown unknown
I1210 10:10:19.334017 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.1 02 unknown unknown
I1210 10:10:19.335102 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.2 02 unknown unknown
I1210 10:10:19.335192 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.3 02 unknown unknown
I1210 10:10:19.335254 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.4 02 unknown unknown
I1210 10:10:19.335316 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.5 02 unknown unknown
I1210 10:10:19.335373 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.6 02 unknown unknown
I1210 10:10:19.335435 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0a.7 02 unknown unknown
I1210 10:10:19.335492 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0b.0 02 unknown unknown
I1210 10:10:19.335594 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:00:0c.0 02 Red Hat, Inc. Virtio network device
I1210 10:10:19.335626 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.0 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335656 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.1 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335668 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.2 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335680 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.3 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335692 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.4 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335704 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.5 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335717 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.6 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335731 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:01:00.7 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335742 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:02:00.0 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335770 1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:10:00.0 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335796 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.0 02 unknown unknown
I1210 10:10:19.335811 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.1 02 unknown unknown
I1210 10:10:19.335818 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.2 02 unknown unknown
I1210 10:10:19.335822 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.3 02 unknown unknown
I1210 10:10:19.335827 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.4 02 unknown unknown
I1210 10:10:19.335832 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.5 02 unknown unknown
I1210 10:10:19.335837 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.6 02 unknown unknown
I1210 10:10:19.335841 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0a.7 02 unknown unknown
I1210 10:10:19.335844 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0b.0 02 unknown unknown
I1210 10:10:19.335847 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:00:0c.0 02 Red Hat, Inc. Virtio network device
I1210 10:10:19.335850 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.0 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335855 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.1 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335858 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.2 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335861 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.3 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335865 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.4 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335869 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.5 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335873 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.6 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335876 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:01:00.7 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335879 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:02:00.0 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335886 1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:10:00.0 02 Red Hat, Inc. Virtio 1.0 network device
I1210 10:10:19.335892 1 main.go:68] Initializing resource servers
I1210 10:10:19.335896 1 manager.go:117] number of config: 1
I1210 10:10:19.335900 1 manager.go:121] Creating new ResourcePool: sriov_dpu
I1210 10:10:19.335903 1 manager.go:122] DeviceType: netDevice
I1210 10:10:19.336001 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.1. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1210 10:10:19.336256 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.2. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1210 10:10:19.336504 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.3. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1210 10:10:19.336782 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.4. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1210 10:10:19.337072 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.5. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1210 10:10:19.337408 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.6. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1210 10:10:19.337750 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:00:0a.7. error getting devlink device attributes for net device 0000:00:0a.0 no such device
I1210 10:10:19.338911 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.1. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1210 10:10:19.339158 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.2. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1210 10:10:19.339575 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.3. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1210 10:10:19.345010 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.4. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1210 10:10:19.349783 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.5. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1210 10:10:19.355430 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.6. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1210 10:10:19.355765 1 utils.go:81] Devlink query for eswitch mode is not supported for device 0000:01:00.7. error getting devlink device attributes for net device 0000:01:00.0 no such device
I1210 10:10:19.362164 1 manager.go:138] initServers(): selector index 0 will register 0 devices
I1210 10:10:19.362186 1 manager.go:142] no devices in device pool, skipping creating resource server for sriov_dpu
I1210 10:10:19.362191 1 main.go:74] Starting all servers...
I1210 10:10:19.362196 1 main.go:79] All servers started.
I1210 10:10:19.362199 1 main.go:80] Listening for term signals