iperf
iperf copied to clipboard
esnet
UDP out of order incoming packet
Iperf3 version : 3.7 OS: Centos Workload ran: -u -b 999M -w 416K -l 1472 Client and server are in same network Iperf3 server is flooded with this messages.
Following is the server output:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-60.00 sec 6.67 GBytes 955 Mbits/sec 0.009 ms 48/4863957 (0.00099%)
[ 5] 60.00-120.00 sec 6.68 GBytes 957 Mbits/sec 0.015 ms 51/4876174 (0.001%)
[ 5] 120.00-180.00 sec 6.68 GBytes 957 Mbits/sec 0.010 ms 26/4875278 (0.00053%)
[ 5] 180.00-240.00 sec 6.69 GBytes 957 Mbits/sec 0.021 ms 63/4876596 (0.0013%)
[ 5] 240.00-300.00 sec 6.69 GBytes 957 Mbits/sec 0.003 ms 69/4876649 (0.0014%)
iperf3: OUT OF ORDER - incoming packet = 28979725 and received packet = 28979750 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 28979726 and received packet = 28979750 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 28979727 and received packet = 28979750 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 28979728 and received packet = 28979750 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 28979729 and received packet = 28979750 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 28979730 and received packet = 28979750 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 28979731 and received packet = 28979756 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 28979732 and received packet = 28979756 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 28979733 and received packet = 28979756 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 28979734 and received packet = 28979756 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 28979735 and received packet = 28979756 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 28979736 and received packet = 28979756 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 28979737 and received packet = 28979762 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 28979738 and received packet = 28979762 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 28979739 and received packet = 28979762 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 28979740 and received packet = 28979762 AND SP = 5
It's odd that the warning message in the bug report is only emitted if the debug flag is set, and it doesn't appear with the same wording in the iperf-3.7 sources. In fact that wording hasn't appeared since the 3.1.x releases. This doesn't look like current code...can you double-check to see what version you're running with iperf3 --version?
Anyway. This error message gets printed if the sequence number for an incoming packet is less than the count of packets received. Basically the sequence numbers jumped backwards. It's an indication of some packets getting misordered (or I suppose duplicated packets could cause this too, but I don't see any sign of this in your output).
Based on what I've seen of the output, you really should check to be sure you're running relatively recent code (3.7 is current as of this writing). There have been a number of improvements in how UDP tests are performed and reported.
ok,I checked the version and here is copied output of the command iperf3 --version iperf 3.7 (cJSON 1.5.2) I have downloaded the source for iperf3 from https://downloads.es.net/pub/iperf/iperf-3.7.tar.gz. The version is released on 2019-06-21.
What can be possible reasons for packet misorder? Can we do something for reducing the same?
Regardless of this issue, it seems that there is a minor bug the the current "OUT OF ORDER" error message. In:
fprintf(stderr, "OUT OF ORDER - incoming packet sequence %" PRIu64 " but expected sequence %d on stream %d", pcount, sp->packet_count, sp->socket);
"sp->packet_count" should probably be "sp->packet_count+1", as the message text say this is the expected packet number (and not the last received).
Regarding the issue, assuming that the problem is not related to the iperf3 version, it seems like somehow two "sub-streams" of 6 packets blocks are received. Once the jitter happened, the read blocks are alternating between these "sub-streams". If this is correct, then the out of order should have stopped when the delayed "sub-stream" reached packet 28979750. Is that what happened? (Of course new jitter and out of order may happen later.)
SP value does not change, It is always 5.(see attached logs for other runs I tried)
Regarding the issue, assuming that the problem is not related to the iperf3 version, it seems like somehow two "sub-streams" of 6 packets blocks are received. Once the jitter happened, the read blocks are alternating between these "sub-streams". If this is correct, then the out of order should have stopped when the delayed "sub-stream" reached packet 28979750. Is that what happened? (Of course new jitter and out of order may happen later.)
-- I see, unfortunately this is not happened.It does not stopped. I am attaching some output from iperf3 which will clarify this (another run).
I ran the workload minimizing the packet size as well but no luck (logs attached).
iperf3out_reducedpacketsize.txt
The two logs are two separate cases: The iperf3_pkt_repeated.txt issues is an overflow in the 32 bits packets counter (note that the received packet 2147483647 = 0x7fffffff). You should use 64 bits packets counter by setting the parameter "--udp-counters-64bit"
The iperf3out_reducedpacketsize.txt issue on the other hand is similar to the initial issue. In this case, it is quite clear that the read packets are alternating between two separate "sub-streams" or two queues. Except for relatively minor reorder (compared to the bits rate) it doesn't seem that there is a real issue. Can it be that in the network the server may receive the packets from the client through two different paths? Are there two network interfaces/cards in the server that the packets may be received through both?
The iperf3_pkt_repeated.txt issues is an overflow in the 32 bits packets counter (note that the received packet 2147483647 = 0x7fffffff). You should use 64 bits packets counter by setting the parameter "--udp-counters-64bit"
--Sure
The iperf3out_reducedpacketsize.txt issue on the other hand is similar to the initial issue. In this case, it is quite clear that the read packets are alternating between two separate "sub-streams" or two queues. Except for relatively minor reorder (compared to the bits rate) it doesn't seem that there is a real issue. Can it be that in the network the server may receive the packets from the client through two different paths? Are there two network interfaces/cards in the server that the packets may be received through both?
-- there are more network interfaces/cards but we are using only one.As we provide server address it will link to associated network interface right ?
Is the problem still there? I also have this problem with streaming, and the bandwidth of UDP is much smaller than TCP. Is there any way to tune it
@zhangtongjian2020 can you please attach log files for he issue you have? It is not clear what is causing this problem and also the issue you have may be different (but with similar results).
Is the problem still there? I also have this problem with streaming, and the bandwidth of UDP is much smaller than TCP. Is there any way to tune it
Hi, I seem to have a similar issue. I have connected two FPGA boards with Gigabit Ethernet directly. Everything seems stable when I try iperf3 using TCP, and I get a maximum BW of close to 4.2 Gbps. However, when I switch to UDP, the maximum BW drops to 3.82, and I get OUT OF ORDER delivery of packets. These OUT OF ORDER deliveries seem to happen in short bursts. When I lower the bitrate cap, the frequency of these bursts happening goes down. Also, the size of the bursts shrinks. I understand that since I am using FPGA boards that I have programmed, it is very likely that the source of this issue is from within my designs and not from the tool, but I wonder if you can help me narrow down my search. Thank you in advance, Arman Client-UDP-3G.txt Client-UDP-5G.txt Server-3G-UDP.txt Server-UDP-5G.txt
@Areman66, it seems that the bitrate throughput is limited by the the CPU performance on the client side. This is because that UDP packets throughput is limited only the by the -bsetting. Therefore, if the client could send the 5Gbps and only 2.5Gbps could be received, the server would show about 50% loss rate. However, (almost) no packet is lost.
You can try to increase the UDP rate by trying to set different packet sizes using the -l option. E.g. the default TCP packet size is 128K and the default UDP packet is about 1.5K, which may be one of the reasons for the performance difference. You can also try to increase the sending burst count. E.g. set the burst count to 20 by -b 5G/20.
As you get the out of order issue only with the 5G network, it is probably related to the 5G network architecture.
By the way, it seems that you are using an old version of iperf3. With the current iperf3 version, the OUT OF ORDER message should have been "OUT OF ORDER - incoming packet sequence ..... but expected sequence ... on stream ...".
Thank you @davidBar-On for your prompt response. I used yum to install iperf3. Should I build it from source for the newer version? Also, regarding the "OUT OF ORDER" issue, as I am using a direct link between the client and the server, I don't think the network infrastructure is causing any out of order delivery of packets. Can I trust the iperf3 not to have caused that and focus on the FPGA design and the driver to find the source of that?
... as I am using a direct link between the client and the server
My mistake. I thought that #G and 5G means cellular network version that you are using for the network connection ....
I used yum to install iperf3. Should I build it from source for the newer version?
I believe it will help to build a new version. At least it will help to make sure that there are no issues because of the old version of iperf3.
What seems strange is the exact 2.5Gbps throughput. Did you try using different packet lengths (using the -l option)? The results of such tests may give more information about the issue.