iperf
iperf copied to clipboard
esnet
UDP packets on dual-homed iperf3 server go out the wrong interface
Context
iperf3 3.1.3 Debian Linux 9.1 x86_64 Server has multiple IPs 192.168.112.61 and 192.168.93.141
Bug Report
Server replies with the first IPv4, while should with the one requested Sample tcpdump: 11:44:49.307320 IP 192.168.93.141.5201 > 10.78.243.11.58002: Flags [P.], seq 2:3, ack 184, win 59, options [nop,nop,TS val 1013109869 ecr 3431431813], length 1 11:44:49.538440 IP 10.78.243.11.54513 > 192.168.93.141.5201: UDP, length 4 11:44:49.538593 IP 192.168.112.61.5201 > 10.78.243.11.54513: UDP, length 4
Server has IPs 192.168.93.141 and 192.168.112.61 Client IP 10.78.243.11
Client is running iperf3 -c 192.168.93.141 -u -b150m -w32M -t60 --get-server-output Session hangs, since UDP reply goes from 192.168.112.61
Expected behavior: Server replies from 192.168.93.141
A possible way is to configure route to client (10.78.243.11) on server preferred source ip 192.168.93.141. Try this. https://superuser.com/questions/376667/how-to-route-only-specific-subnet-source-ip-to-a-particular-interface
I had to draw myself a picture to see what's going on. But it appears the problem is that outbound UDP traffic from the server isn't going out through the same interface that it came in.
The OP didn't say anything about how routing is set up, but something along the lines of what @dezenxi suggested might be helpful as a workaround.
I'd like to understand why UDP packets went out a different interface. I haven't looked at the code yet, but this does seem wrong or at least non-intuitive. I'm going to say this is a real bug, at least for now.
@afpd: TCP tests work for you, correct? Can you show me the output if netstat -rn on the server so I can see the kernel routing table? Also note that you're running an old version of iperf3. 3.2 is current as of this writing.
Server has one NIC, although lo (loopback) has 192.168.93.141/32, while eth0 has 192.168.112.61/24.
@afpd: Thanks for the additional information, although having answers to the rest of my questions would be useful too.
Is this bug resolved in the latest version? For some reason, I can't install the latest right now. But information on whether it is fixed or not would be helpful. Thanks
@ks228: No coding changes have been done towards fixing this issue. Indeed at this point I'm not sure exactly what the issue is, or whether it's a bug in iperf3 or in the OP's setup. Still waiting on answers to some of the follow-up questions I asked.
@bmah888 I actually tested it on my setup just a few minutes ago on versions 3.1.3 and the latest 3.3. It is an issue in my setup as well. My setup is simple. I have multiple addresses configured on my server's NIC. I second that it only happens with UDP. TCP works fine here as well. There is this thread as well but that solution doesn't work https://github.com/esnet/iperf/issues/371#issuecomment-213584658 at least in my simple setup.
I'm seeing this issue, as well.
On server
pi@wolfepi:~ $ iperf3 -s -V
iperf 3.0.7
Linux wolfepi 4.9.35-v7+ #1014 SMP Fri Jun 30 14:47:43 BST 2017 armv7l GNU/Linux
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Time: Tue, 23 Jan 2018 04:33:24 GMT
Accepted connection from 208.44.206.101, port 45776
Cookie: jumpbox.1516682001.727054.3c8ab9de6a
[ 5] local 10.2.0.70 port 5201 connected to 208.44.206.101 port 54888
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 10 second test
Server hangs until I Ctrl-C
^C[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-542.39 sec 0.00 Bytes 0.00 bits/sec 0.000 ms 0/0 (nan%)
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-542.39 sec 0.00 Bytes 0.00 bits/sec 0.000 ms 0/0 (nan%)
CPU Utilization: local/receiver 0.0% (0.0%u/0.0%s), remote/sender 0.0% (0.0%u/0.0%s)
iperf3: interrupt - the server has terminated
On client
C:\Users\rwolfe\Downloads\iperf-3.1.3-win64>iperf3.exe -c xxxxx-iperf.ddns.net -
u -b 500000000
Connecting to host wolfe-iperf.ddns.net, port 5201
Client hangs until I Ctrl-C
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datag
rams
iperf3: interrupt - the client has terminated
Note that the client has terminated only results when I Ctrl-C the process.
More Info
Server's NIC Info
pi@wolfepi:~ $ ifconfig
eth0 Link encap:Ethernet HWaddr b8:27:eb:0e:cf:10
inet addr:10.2.0.70 Bcast:10.2.0.255 Mask:255.255.255.0
inet6 addr: fe80::470c:ff13:a6d0:afd8/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:103276 errors:0 dropped:1 overruns:0 frame:0
TX packets:56242 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:133280154 (127.1 MiB) TX bytes:9376483 (8.9 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:200 errors:0 dropped:0 overruns:0 frame:0
TX packets:200 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:16656 (16.2 KiB) TX bytes:16656 (16.2 KiB)
wlan0 Link encap:Ethernet HWaddr b8:27:eb:5b:9a:45
inet addr:10.2.0.71 Bcast:10.2.0.255 Mask:255.255.255.0
inet6 addr: fe80::3c8f:1bd3:dad0:9802/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:586 errors:0 dropped:0 overruns:0 frame:0
TX packets:90 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:106403 (103.9 KiB) TX bytes:16860 (16.4 KiB)
Server's routing table
pi@wolfepi:~ $ netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.2.0.1 0.0.0.0 UG 0 0 0 eth0
0.0.0.0 10.2.0.1 0.0.0.0 UG 0 0 0 wlan0
10.2.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.2.0.0 0.0.0.0 255.255.255.0 U 0 0 0 wlan0
The initial connection comes in through the wlan0 interface, but the UDP responses seem to go out of the eth0 interface which then hangs.
For what it's worth, TCP does work. Both sides are behind a firewall with NAT, and I own the network on both sides, so I can do some tweaking as needed.
Let me know if I can test anything out for you, @bmah888. I'm happy to help get this resolved.
After writing my above comment, I realized that I was accidentally sending the iperf requests to the wlan0 interface. This was accidental. I fixed it to request to the eth0 interface and UDP began working. Presumably because the return flow was going back out of the same interface.
It doesn't necessarily fix the issue; however, it does seem to suggest that the OS is prioritizing NICs and could be impacting iperf UDP testing since the UDP test actually receives a request to the server, and then starts a new UDP stream to the client.
Somewhat similar to how Active mode FTP opens up a new data channel after the control channel is established.
Hopefully this sheds some light, but, regardless, let me know if I can test anything out for you.
Workaround:
I have a single linux virtual NIC with multiple Ips assigned to it. Iperf3 does not work when u try to generate traffic to multiple IPs on that NIC. BUT you can start servers while binding those processes to the IP that you want and that seems to work
Can reproduce this issue with iperf 3.7 (on a host with 2 NICs that have IP addressed assigned from the same subnet), i.e. the pair iperf3 -c $sever_addr_2nd -u and iperf -s yield
iperf3: error - unable to read from stream socket: Connection refused
on the server and the client can't start to send.
I can also confirm hkominos' workaround, i.e. to let the server bind to the right interface:
iperf3 -s -B $server_addr_2nd
I ran into this today. After digging around far too much, I think this is an iperf bug. Or at least, iperf is behaving in a way I found surprising.
The issue is in these lines of code in iperf_udp.c. When iperf calls connect on the remote address, this binds the socket to use a source address automatically selected by the system's routing table. In systems with more than one globally-routable address, usually there are routes predicated on the source address, but in this case there is no source address set yet and those rules can't fire. This results in iperf automatically binding to a different address than the original packet came in on.
I was expecting iperf to reply to packets using the same address. It does so for TCP, but not for UDP. This was unexpected for me, but during testing I learned that a lot of programs (even netcat!) have the same behavior.
hkominos's workaround works because if you bind the socket to an address ahead of time, outgoing packets are forced to use the bound address.
If this is unintended behavior, the fix would be to read the destination address used in the initial packet and then bind to that address manually before connect. Reading this link, it's no surprise so few UDP programs bother to do this -- the APIs don't make it easy to get this info, and probably nobody tests their code on multi-home systems.
If this is unintended behavior, the fix would be to read the destination address used in the initial packet and then bind to that address manually before
connect. Reading this link, it's no surprise so few UDP programs bother to do this -- the APIs don't make it easy to get this info, and probably nobody tests their code on multi-home systems.
I ran into this as well. The issue is indeed as you describe it. The iperf server opens a UDP socket, binds it to whatever it has to listen for incoming packets on, then when it receives the first packet from the client, it connects that socket to the client's address/port as reported by recvfrom. After that, it opens and binds a new socket to be used for new incoming packets, while keeping the previous socket for this session.
I have managed to write a quick-and-dirty fix. The only reliable way to make the kernel use the right source address is to bind the socket to it, but the current session socket has already been bound (and re-binding is forbidden). So a new socket has to be created, bound to the right source address (taken from the ip_pktinfo/ip6_pktinfo ancillary data) and connected to the client's address and port.
While this approach works, it is certainly not a fix satisfactory enough for the maintainers, as it is not satisfactory enough for me at least. Nevertheless, if you're interested, I can make a pull request.
Just to chime in and say, I just ran into this issue as well using iperf3 version 3.14 as the server. I had to -B bind to the interface I needed.
@iazz,
While this approach works, it is certainly not a fix satisfactory enough for the maintainers, as it is not satisfactory enough for me at least.
I am not from the maintenance team, but I am interested to understand why you don't believe you fix is not satisfactory. Is it just because it is a kind of a prototype for the real fix? Is it too complex? Other?
@iazz,
While this approach works, it is certainly not a fix satisfactory enough for the maintainers, as it is not satisfactory enough for me at least.
I am not from the maintenance team, but I am interested to understand why you don't believe you fix is not satisfactory. Is it just because it is a kind of a prototype for the real fix? Is it too complex? Other?
Quite frankly, it's been a long time and I can't remember. At first glance, the fix seems quite alright. Maybe it's that this won't work on all platforms. At least it works on Linux.
You can clone it off my repository and give it a shot, if you're interested.