kvm-guest-drivers-windows
kvm-guest-drivers-windows copied to clipboard
virtio-win
netkvm: For tunnel protocols like GRE, support the inner layer IP and port as elements of the RSS five-tuple algorithm
Is your feature request related to a problem? Please describe. In tunnel protocols like GRE, the outer layer protocol's IP and port are always the same, resulting in network packet reception being consistently scheduled on the same CPU.This can lead to a decrease in network reception efficiency
Describe the solution you'd like For tunnel protocols like GRE, is it possible for the RSS five-tuple algorithm to support the inner layer IP and port as elements of the five-tuple?
Additional context Currently, I have modified the code in the driver to support the inner layer IP and port for the GRE protocol. The effect is quite significant as our QPS has increased from 30000 to 140000. Therefore, I am considering whether we can consider this as our baseline solution.
@zjmletang Is your work based on the virtio-net spec changes on the matter? If yes, please share the code.
I couldn't find in the spec（https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html#x1-2040005） any specification regarding whether the RSS five-tuple algorithm should use the inner IP and port or the outer IP and port.
Currently, in my work, for tunneling protocols (only supporting GRE), we are fixedly using the inner IP and port. However, I am unsure if this is compliant with the specification.
@zjmletang The existing spec defines RSS hash calculation the same way as WDK defines it, i.e. for outer IP:port. Latest addition might include additional hash types, I remember some discussion at least. But if we discuss the feature (GRE/inner hash calculation) let's start from a) GRE support under Windows, b) use case, c) test setup
@zjmletang The existing spec defines RSS hash calculation the same way as WDK defines it, i.e. for outer IP:port. Latest addition might include additional hash types, I remember some discussion at least. But if we discuss the feature (GRE/inner hash calculation) let's start from a) GRE support under Windows, b) use case, c) test setup
@ybendito Could you please guide me on where to find information regarding the use of outer IP and port for GRE (Generic Routing Encapsulation)? I have been searching but haven't found any documentation on this matter. Even though I've reviewed the hash types on https://learn.microsoft.com/en-us/windows-hardware/drivers/network/rss-hashing-types, it doesn't seem to specify whether GRE utilizes the outer or inner IP/port.
@zjmletang AFAIR, with limited responsibility: The GRE packet is the IP packet with its own header, so it uses its own IP. Term "port" might be not available in case of GRE (it could be IP with its own type or UDP or TCP as the PPTP is also kind of GRE) and the inner might be something completely different, so regarding the IP - it shouldn't be the same as inner and port should be also different or absent. I suspect that the inner header (if the inner is IP) might be encrypted. May be you can find something here https://packetlife.net/captures/protocol/gre/
@ybendito ，The following is for discussion purposes only, as I have limited knowledge of the Linux kernel and there may be inaccuracies. I've noticed that Linux kernel's RPS seems to support parsing the inner protocols of GRE. To my understanding, in Linux, the RSS function is handled within the kernel. In contrast, for Windows, RSS is implemented within drivers. Correspondingly, if RPS in Linux supports GRE, then for our Windows drivers, there might be a necessity to support parsing the inner protocols of GRE as well. Perhaps we don't need to support all inner GRE protocols, but just focus on supporting scenarios where the inner protocols are UDP, TCP, and the like.
@zjmletang Let's focus on Windows. The full responsibility of the driver is a) to report incoming packets to the proper queue according to OS expectations and b) to report the calculated hash of each packets in the packet's metadata. If the device is able to deliver packets to proper queues (such a way that HCK tests pass) and provide the calculated hash with each packet (according to virtio spec VIRTIO_NET_F_RSS and VIRTIO_NET_F_HASH_REPORT) the driver can skip all other processing ; this will require some change in the driver but this is possible and this is a roadmap. In case of virtio-net on QEMU there are some gaps at the moment, but the direction is as I've described. In case of hardware devices - I do not have any information about the capabilities of your device, so I do not know whether there are gaps or not. Now with limited understanding as I did not study this area enough: GRE: At the moment the netkvm driver does not declare support of the GRE to Windows and does not support it, this means both RX direction (RSS and checksum validateion) and TX direction (LSO and checksum offloads). Support for GRE involves also dedicated certification tests.
@ybendito Okay, thank you. I've understood your point, and indeed, I also think the roadmap you mentioned is the more reasonable one.