rr icon indicating copy to clipboard operation
rr copied to clipboard
verified publisher icon RROrg

The rr latest versions don't conduct the internet connection correctly for NIC sriov vf iavf driver... intel万兆网卡sriov虚拟化后vf直通黑群晖不能连接。

Open bbadm1108 opened this issue 1 year ago • 8 comments

请填写以下信息.
Please fill in the following information.

Install ENV: (You can find it in the boot interface.)

  • DMI: PVE 8.2
  • CPU: intel i9-10900t
  • NIC: (pid & vid) 0x46f4 / 0x0001
  • intel x710 DA2

RR version: (You can find it in the update menu.)

  • RR: 24.4.x-24.6.x
  • addons:
  • modules: iavf
  • lkms:

DSM:

  • model: DS920plus and SA6400 etc
  • version: 7.2.1
  • vid/pid: 0x46f4 / 0x0001

Issue:

PVE has set up the intel x710DA2 nic card in sr-iov mode and generated 6 vfs for each pf, but vf don't work at all in DSM 920+/SA6400 vm(s) with the booting message "eth1(iavf)" " "LINK LOCAL, NO DHCP SERVER DETECTED"..

  1. All the vf(s) work fine in windows/openwrt VM(s) by passthrough, but not in DSM vm.
  2. If I pass through the pf(not vf) directly to rr-DSM vm, it boots up perfectly with driver 'i40e'.
  3. the message greped in PVE dmesg: [ 410.696718] i40e 0000:01:00.0: VF attempting to override administratively set MAC address, bring down and up the VF interface to resume normal operation

Call for help to fix the problem, thanks

PVE在sr-iov模式下设置了intel x710DA2万兆网卡，并为每个pf生成了6个虚拟网卡vfs。但vf在rr/DSM 920+/SA6400不能联网，引导消息为“eth1(iavf)”。"LINK LOCAL, NO DHCP SERVER DETECTED"…

1）所有的vf都可以直通其它虚拟机windows/openwrt中正常连通网络。 2）如果将物理网卡pf直通给rr-dsm虚拟机，可以完美地启动连通，驱动为'i40e'，只有vf在'iavf'驱动下失连。 3）PVE控制台dmesg|grep MAC显示：[ 410.696718] i40e 0000:01:00.0: VF attempting to override administratively set MAC address, bring down and up the VF interface to resume normal operation

呼叫帮助，谢谢！

logs:

(请先看一下#173、#175、#22 / /6的内容)
(Plz review the content of #173, #175, #226 first)
...

... 如果你提供不了详细信息，那就等有缘人吧! ... If you can't provide detailed information, then wait for someone who is destined!

bbadm1108 avatar Jun 12 '24 02:06 bbadm1108

1）不是pve网卡sriov虚拟化本身的问题，因为所有vf都能正常直通到其它vm工作，如Windows。 2）试过直通8125和x710物理网卡给群晖虚拟机，都可以正常联网，唯独sriov vf不行。

bbadm1108 avatar Jun 12 '24 08:06 bbadm1108

dmesg | grep iavf

wjz304 avatar Jun 24 '24 08:06 wjz304

SSH to DSM as follow, the NIC is setup and can be seen as 10Gbps in DSM, while the IP address is 169.254.xx.xx that could not be connected.

dmesg | grep iavf

pve@DS920plus:~$ dmesg | grep iavf [ 5.073885] iavf: Intel(R) Ethernet Adaptive Virtual Function Network Driver - version 4.5.3 [ 5.141056] iavf 0000:04:00.0: Invalid MAC address 00:00:00:00:00:00, using random [ 5.141926] iavf 0000:04:00.0: Multiqueue Enabled: Queue pair count = 4 [ 5.142788] iavf 0000:04:00.0: MAC address: 72:2c:f1:62:c9:30 [ 5.143198] iavf 0000:04:00.0: GRO is enabled [ 26.343270] Modules linked in: i915(+) drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops video backlight i2c_algo_bit dm_crypt sg dm_snapshot dm_bufio crc_itu_t crc_ccitt psnap p8022 llc hfsplus zram md4 hmac sit tunnel4 ipv6 flashcache_syno(O) flashcache(O) syno_flashcache_control(O) dm_mod arc4 crc32c_intel cryptd ecb aes_x86_64 authenc des_generic ansi_cprng cts md5 cbc cpufreq_powersave cpufreq_performance cpufreq_stats vxlan ip6_udp_tunnel udp_tunnel ip_tunnel kvm_intel kvm irqbypass loop btrfs raid6_pq xor zstd_compress zstd_decompress ecryptfs xxhash sha256_generic synorbd(O) synofsbd(O) hid_generic usbhid hid virtio_net(OE) etxhci_hcd virtio_pci(OE) virtio(OE) virtio_ring(OE) drm drm_panel_orientation_quirks iosf_mbi cfbfillrect cfbimgblt cfbcopyarea iavf(OE) auxiliary(OE) fb fbdev(OE) [ 27.236586] iavf 0000:04:00.0: Failed to add MAC filter, error Unknown status code (update virtchnl.h) [ 27.241050] iavf 0000:04:00.0 eth0: changing MTU from 1500 to 9000 [ 27.518259] iavf 0000:04:00.0 eth0: NIC Link is Up Speed is 10 Gbps Full Duplex

bbadm1108 avatar Jun 24 '24 14:06 bbadm1108

virtchnl.h 不兼容， 即使 RR 兼容了，DSM也不一定兼容

wjz304 avatar Jun 24 '24 14:06 wjz304

virtchnl.h 不兼容， 即使 RR 兼容了，DSM也不一定兼容

rr引导启动时，vf是可以获取dhcp自动分配一个ip地址，只是passthrough到dsm后，主机vf mac清零为00:00:00:00:00:00，群晖vm内部分配随机mac和local ip地址，从此失联。

bbadm1108 avatar Jun 24 '24 14:06 bbadm1108

https://www.geekxw.top/638/ 试试这个，不知道是不是需要下面那个脚本的原因，只能vf网卡之间互通 我的82599也是用的sriov给群晖分配的虚拟网卡是没问题的

Niko-1118 avatar Jul 07 '24 08:07 Niko-1118

谢谢！

这个脚本曾经试过，只搜索vm添加的桥接网络mac，而不是直通的pcie网卡设备，未能解决vf直通群晖后失联问题。。。我回头再研究一下。

bbadm1108 avatar Jul 09 '24 10:07 bbadm1108

DS920+ / DSM7.2 / rr 24.9.1 今天换了mellanox cx4421a网卡，打开sriov创建vf后直通群晖，问题依旧。

  1. 直通的vf mac地址清零，群晖能发现vf网卡但不能联网。
  2. 直通pf给群晖可以正常驱动联网。

bbadm1108 avatar Oct 08 '24 09:10 bbadm1108

有了进展。 1）因为其它原因重装了pve8.2 2）试了x710DA2和cx4421a两个网卡，其中第一个网口（网口0）可以生成sriov vf，用了最新的rr 24.7.x引导后，vf直通DSM7.2.2-72806可以连通，mac地址也不再清零了。 3）第二个网口（网口1）生成vf失败，只能pf使用。 目前不清楚是哪个环节导致的恢复（pve重装？b460m主板？rr/dsm更新？），目前工作正常，此问题暂时解决。

bbadm1108 avatar Oct 25 '24 04:10 bbadm1108