iperf
iperf copied to clipboard
esnet
iperf3 client server is not sending or receiving any packets for the one port in a two port adapter server
- Version of iperf3: iperf3-3.5-7.el8.x86_64
- Hardware: x86_64
- Operating system (and distribution, if any): Ubuntu 22.04 and iperf3 server OS is RHEL8.6
Bug Report
*** Expected Behavior**
- iperf3 has to send and receive packets through both the ports of network adapter
*** Actual Behavior**
- iperf3 is sending packets through only one port for the second port 0bytes are transferred
*** Possible Solution**
- Packets has to send from both the ports
Enhancement Request
I am trying to do network stress on Ubuntu 22.04 OS and Iperf3 server has RHEL8.6 OS
On iperf3 server:
- Disabled the firewall
- Installed iperf3 package
- Made it as iperf3 server by giving command -> #iperf3 -s
On Ubuntu 22.04 OS
- My server has 2 ports NIC adapter.
- In the OS, disabled firewall.
- When the test runs, for the first port it is giving enough bandwidth and for second port packets are dropping to 0 bytes.
- Test is designed in such a way by OS that it considers iperf3 tool only. When the first port is running second port will be down, once first port network completes testing 2nd port will be up and starts running the network test. Everything is automated in the script, so we can't interfere in-between to change the ports.
- Current behavior
- Packets transferring is happening to one port, other port is completely blocked
- Desired behavior
- Packets has to transfer from both ports.
- Below is the result of iperf3 -s of both the ports
[root@123 ~]# iperf3 -c 10.1.1.2
Connecting to host 10.1.1.2, port 5201
[ 5] local 10.1.1.1 port 40396 connected to 10.1.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 2.32 GBytes 19.9 Gbits/sec 0 1.14 MBytes
[ 5] 1.00-2.00 sec 2.33 GBytes 20.0 Gbits/sec 0 1.14 MBytes
[ 5] 2.00-3.00 sec 2.34 GBytes 20.1 Gbits/sec 0 1.14 MBytes
[ 5] 3.00-4.00 sec 2.34 GBytes 20.1 Gbits/sec 0 1.14 MBytes
[ 5] 4.00-5.00 sec 2.34 GBytes 20.1 Gbits/sec 0 1.14 MBytes
[ 5] 5.00-6.00 sec 2.34 GBytes 20.1 Gbits/sec 0 1.14 MBytes
[ 5] 6.00-7.00 sec 2.34 GBytes 20.1 Gbits/sec 0 1.14 MBytes
[ 5] 7.00-8.00 sec 2.34 GBytes 20.1 Gbits/sec 0 1.14 MBytes
[ 5] 8.00-9.00 sec 2.34 GBytes 20.1 Gbits/sec 0 1.14 MBytes
[ 5] 9.00-10.00 sec 2.34 GBytes 20.1 Gbits/sec 0 1.14 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 23.4 GBytes 20.1 Gbits/sec 0 sender
[ 5] 0.00-10.04 sec 23.4 GBytes 20.0 Gbits/sec receiver
iperf Done.
[root@123 ~]# iperf3 -c 10.1.1.3
Connecting to host 10.1.1.3, port 5201
[ 5] local 10.1.1.1 port 54716 connected to 10.1.1.3 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 323 KBytes 2.65 Mbits/sec 7 8.74 KBytes
[ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
[ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 323 KBytes 265 Kbits/sec 10 sender
[ 5] 0.00-10.04 sec 0.00 Bytes 0.00 bits/sec receiver
iperf Done.
What can be wrong here? How we can make other port get packets from iperf3 client?
I guess that the problem is related to the Cwnd size of the second port which is 8.74 KBytes v.s. 1.14 MBytes for the first one. Cwnd is the TCP tcpi_snd_cwnd value that determines the congestion window (see here). Probably this port's cwnd size should be increased.
The first 323 KBytes sent in the first interval may be just the data transferred to the client's buffers (or switches/routers in the way to the server), not the actual amount arrived to the server. One possibility is that the 323 KBytes are still slowly being sent to the server. You should check the server report.
My test environment is of 100G, the switch is 100G and cables I have used is also 40G. So hardware side we are good. This issue we're seeing whenever we use to test the speed which is more than 20G, 40G, 100G. We fail to achieve the 75% of the bandwidth it always falls under 40%. How can I increase the cwnd size? any command to do that?
We fail to achieve the 75% of the bandwidth it always falls under 40%.
With iperf3 of high speed network, the limit may be client and/or server machine CPU. Check the CPU usage when the test is running, e.g. using top. If the CPU utilization is high, you may try building and testing the beta multi-thread version of iperf3, which performs better for high speed networks.
How can I increase the cwnd size? any command to do that?
Reading more about CWND, it is set by the congestion control algorithm. See for example Wikipedia page about congestion control. Therefore I guess that there is a problem in the interface using the 2nd port. Wireshark (and/or tcpdump) may be required to analyze the traffic between the client and server to better understand the issue.
It is also important to understand what congestion control algorithm is used and if it is the same for both ports. If you run iperf3 with the -V option, it prints the algorithm used at the end of the test. Is it the same for both ports?
You can also try running UDP tests, to see whether the issue is related to TCP or is problem may be with the 2nd port. -u option is used to run UDP tests. Try running udp test on the 2nd port with low bandwidth first, e.g. 10Kbits/sec by setting -b 10K. If it works ok, increase the bitrate in each test (e.g. 100K, 1M, 100M. 1G, 10G), until you reach a bandwidth with many lost packets. This will indicate the throughput that can be achieved with the 2nd port. (By the way, you may try these different bit-rate tests also with TCP and see what happens.)
To see the CWND related values used during the test, use ss -it. The relevant TCP sockets info will be with one port 5201 and the other is the port reported by iperf3. E.g., on my computer I get (client and server sockets, as I run both client and server on the same device):
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
ESTAB 0 2698540 127.0.0.1:43366 127.0.0.1:5201
cubic wscale:7,7 rto:210 rtt:0.035/0.014 mss:65483 pmtu:65535 rcvmss:536 advmss:65483 cwnd:42 ssthresh:38 bytes_sent:30247408377 bytes_acked:30247408378 segs_out:555019 segs_in:226998 data_segs_out:555017 send 628636.8Mbps lastrcv:4020 pacing_rate 741129.7Mbps delivery_rate 116414.2Mbps delivered:555018 busy:4020ms rwnd_limited:2700ms(67.2%) reordering:4 reord_seen:2 rcv_space:65495 rcv_ssthresh:65495 notsent:2698540 minrtt:0.001
ESTAB 5727885 0 [::ffff:127.0.0.1]:5201 [::ffff:127.0.0.1]:43366
cubic wscale:7,7 rto:210 rtt:0.005/0.002 ato:40 mss:32768 pmtu:65535 rcvmss:65483 advmss:65483 cwnd:10 bytes_received:30281852435 segs_out:227160 segs_in:555545 data_segs_in:555543 send 524288.0Mbps lastsnd:4030 lastack:110 pacing_rate 1048576.0Mbps delivered:1 rcv_rtt:0.019 rcv_space:4391071 rcv_ssthresh:3145728 minrtt:0.005
Can you send this info from your test (logged while the test is running)?
Note that in my case "cubic" is the congestion control algorithm, advmss*cwnd (65483*42) is the cwnd iperf3 reports (about 2.5MB), bytes_sent/bytes_received and bytes_acked indicates how much data was received on the server, etc.
What's the interface address configuration of the two interfaces on your server machine (RHEL)?
Just looking at the addresses, it looks like you have both of those interfaces addressed on the same subnet. If so, that's slightly unusual (some might say "broken") and might be causing some oddness in the IP routing on the server host.
On your client machine (Ubuntu), does it have one network interface or two? What's the physical network topology look like between these two machines and their network interfaces?
I have run into a similar issue before. Along the same lines as what Bruce is saying about the same subnet, the Linux networking stack will behave in unexpected ways when you have two links on the same subnet. One thing that might be happening here is both interfaces are responding to ARPs for all IP addresses regardless of the interface's IP address. See arp_ignore https://sysctl-explorer.net/net/ipv4/arp_ignore/
I would guess that the working one has a better routing metric for that subnet if you were to check ip route.
One indicator that this is happening is using the --bind flag i.e. iperf3 -s --bind 10.1.1.2%enp0s31f6. A possible work around is running multiple instances of iperf each bound to it's own VRF device, then binding the relevant interfaces to that VRF.
