firecracker
firecracker copied to clipboard
Published
20 hours ago •
firecracker-microvm
firecracker-microvm
Use scatter-gather IO in Firecracker virtio-net TX path
The virtio network device emulation implementation in Firecracker is currently using an intermediate buffer to build the TX packet from the guest virtqueue descriptors and then calls write() on the host tap to send this packet (see NetEpollHandler::process_tx() in devices/src/virtio/net.rs).
When doing write(), the data is again copied in the kernel from the userspace Firecracker intermediate buffer to the kernel tap buffers.
An alternative is to use a scatter-gather IO system call like writev() which would take a list of memory addresses for the packet parts (in this case the addresses of the data portion of the virtqueue descriptors), thus avoiding the intermediate buffer and avoiding an extra step of moving the full packet around in memory.
See https://www.gnu.org/software/libc/manual/html_node/Scatter_002dGather.html
@lyuyuan92 sounds good! Please do some benchmarking as well to make sure this is an improvement.
As an example: check out the Firecracker demo to see how iperf is used for measuring network performance.
Also check out https://github.com/firecracker-microvm/firecracker/pull/937 to see another improvement and how it was benchmarked.
Rust nightly provides a method called write_vectored in the std::io::Write trait. Since Tap implements Write trait, we could use the default write_vectored implementation which calls write on the first nonempty buffer provided.
Edit: Never mind, I guess the point of this issue is really avoiding using any kinds of buffer and write from the guest memory directly?
@lyuyuan92 indeed, the point of it (if yet possible in rust) is to avoid buffers in Firecracker device emulation code and write from the virtio rings (guest memory as you say) directly.
Also note that we are limited to using stable features of rust.
@lyuyuan are you still interested in this issue ? If not I would like to take it.
@lyuyuan are you still interested in this issue ? If not I would like to take it.
Hi! Please go ahead and take it. I have worked on it but got stuck having the writev to work. I'd like to learn by reading your solution!
Here is a working example.
The problem is that I don't know if we'll still be able to emit the tx_spoofed_mac_count metric this way.
I worked on a new PoC that emits tx_spoofed_mac_count and tried to tweak it a bit. Here it is. Unfortunately it seems to be slower then before. Here are the results:
without scatter-gather IO:
root@u408d5cb7e82359d64def:~# iperf3 -c 172.16.0.1 -t 60
Connecting to host 172.16.0.1, port 5201
[ 4] local 172.16.0.2 port 43248 connected to 172.16.0.1 port 5201
[ 49.266636] random: crng init done
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 2.16 GBytes 18.6 Gbits/sec 0 928 KBytes
[ 4] 1.00-2.00 sec 2.22 GBytes 19.1 Gbits/sec 0 928 KBytes
[ 4] 2.00-3.00 sec 2.15 GBytes 18.4 Gbits/sec 0 928 KBytes
[ 4] 3.00-4.00 sec 2.19 GBytes 18.9 Gbits/sec 0 928 KBytes
[ 4] 4.00-5.00 sec 2.21 GBytes 18.9 Gbits/sec 0 928 KBytes
[ 4] 5.00-6.00 sec 2.16 GBytes 18.6 Gbits/sec 0 928 KBytes
[ 4] 6.00-7.00 sec 2.23 GBytes 19.1 Gbits/sec 0 928 KBytes
[ 4] 7.00-8.00 sec 2.31 GBytes 19.9 Gbits/sec 0 928 KBytes
[ 4] 8.00-9.00 sec 2.27 GBytes 19.5 Gbits/sec 0 928 KBytes
[ 4] 9.00-10.00 sec 2.22 GBytes 19.0 Gbits/sec 0 928 KBytes
[ 4] 10.00-11.00 sec 2.29 GBytes 19.7 Gbits/sec 0 928 KBytes
[ 4] 11.00-12.00 sec 2.14 GBytes 18.4 Gbits/sec 0 928 KBytes
[ 4] 12.00-13.00 sec 2.23 GBytes 19.1 Gbits/sec 0 928 KBytes
[ 4] 13.00-14.00 sec 2.12 GBytes 18.2 Gbits/sec 0 928 KBytes
[ 4] 14.00-15.00 sec 2.20 GBytes 18.9 Gbits/sec 0 928 KBytes
[ 4] 15.00-16.00 sec 2.28 GBytes 19.6 Gbits/sec 0 928 KBytes
[ 4] 16.00-17.00 sec 2.29 GBytes 19.7 Gbits/sec 0 928 KBytes
[ 4] 17.00-18.00 sec 2.21 GBytes 18.9 Gbits/sec 0 928 KBytes
[ 4] 18.00-19.00 sec 2.24 GBytes 19.3 Gbits/sec 0 928 KBytes
[ 4] 19.00-20.00 sec 2.22 GBytes 19.0 Gbits/sec 0 928 KBytes
[ 4] 20.00-21.00 sec 2.24 GBytes 19.2 Gbits/sec 0 928 KBytes
[ 4] 21.00-22.00 sec 2.18 GBytes 18.7 Gbits/sec 0 928 KBytes
[ 4] 22.00-23.00 sec 2.27 GBytes 19.5 Gbits/sec 0 928 KBytes
[ 4] 23.00-24.00 sec 2.31 GBytes 19.8 Gbits/sec 0 928 KBytes
[ 4] 24.00-25.00 sec 2.29 GBytes 19.7 Gbits/sec 0 928 KBytes
[ 4] 25.00-26.00 sec 2.26 GBytes 19.4 Gbits/sec 0 928 KBytes
[ 4] 26.00-27.00 sec 2.20 GBytes 18.9 Gbits/sec 0 928 KBytes
[ 4] 27.00-28.00 sec 2.19 GBytes 18.8 Gbits/sec 0 928 KBytes
[ 4] 28.00-29.00 sec 2.12 GBytes 18.2 Gbits/sec 0 928 KBytes
[ 4] 29.00-30.00 sec 2.25 GBytes 19.3 Gbits/sec 0 928 KBytes
[ 4] 30.00-31.00 sec 2.28 GBytes 19.6 Gbits/sec 0 928 KBytes
[ 4] 31.00-32.00 sec 2.27 GBytes 19.5 Gbits/sec 0 928 KBytes
[ 4] 32.00-33.00 sec 2.19 GBytes 18.8 Gbits/sec 0 928 KBytes
[ 4] 33.00-34.00 sec 2.26 GBytes 19.4 Gbits/sec 0 928 KBytes
[ 4] 34.00-35.00 sec 2.24 GBytes 19.3 Gbits/sec 0 928 KBytes
[ 4] 35.00-36.00 sec 2.22 GBytes 19.1 Gbits/sec 0 928 KBytes
[ 4] 36.00-37.00 sec 2.23 GBytes 19.2 Gbits/sec 0 928 KBytes
[ 4] 37.00-38.00 sec 2.26 GBytes 19.4 Gbits/sec 0 928 KBytes
[ 4] 38.00-39.00 sec 2.26 GBytes 19.4 Gbits/sec 0 928 KBytes
[ 4] 39.00-40.00 sec 2.22 GBytes 19.0 Gbits/sec 0 928 KBytes
[ 4] 40.00-41.00 sec 2.22 GBytes 19.0 Gbits/sec 0 928 KBytes
[ 4] 41.00-42.00 sec 2.24 GBytes 19.2 Gbits/sec 0 928 KBytes
[ 4] 42.00-43.00 sec 2.16 GBytes 18.6 Gbits/sec 0 928 KBytes
[ 4] 43.00-44.00 sec 2.11 GBytes 18.1 Gbits/sec 0 928 KBytes
[ 4] 44.00-45.00 sec 2.21 GBytes 19.0 Gbits/sec 0 928 KBytes
[ 4] 45.00-46.00 sec 2.30 GBytes 19.7 Gbits/sec 0 928 KBytes
[ 4] 46.00-47.00 sec 2.24 GBytes 19.2 Gbits/sec 0 928 KBytes
[ 4] 47.00-48.00 sec 2.17 GBytes 18.6 Gbits/sec 0 928 KBytes
[ 4] 48.00-49.00 sec 2.19 GBytes 18.8 Gbits/sec 0 928 KBytes
[ 4] 49.00-50.00 sec 2.16 GBytes 18.5 Gbits/sec 0 928 KBytes
[ 4] 50.00-51.00 sec 2.11 GBytes 18.1 Gbits/sec 0 928 KBytes
[ 4] 51.00-52.00 sec 2.06 GBytes 17.7 Gbits/sec 0 928 KBytes
[ 4] 52.00-53.00 sec 2.05 GBytes 17.6 Gbits/sec 0 928 KBytes
[ 4] 53.00-54.00 sec 2.10 GBytes 18.0 Gbits/sec 0 928 KBytes
[ 4] 54.00-55.00 sec 2.06 GBytes 17.6 Gbits/sec 0 928 KBytes
[ 4] 55.00-56.00 sec 2.07 GBytes 17.8 Gbits/sec 0 928 KBytes
[ 4] 56.00-57.00 sec 2.08 GBytes 17.9 Gbits/sec 0 928 KBytes
[ 4] 57.00-58.00 sec 2.11 GBytes 18.1 Gbits/sec 0 928 KBytes
[ 4] 58.00-59.00 sec 1.94 GBytes 16.6 Gbits/sec 0 928 KBytes
[ 4] 59.00-60.00 sec 2.15 GBytes 18.5 Gbits/sec 0 928 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.00 sec 132 GBytes 18.9 Gbits/sec 0 sender
[ 4] 0.00-60.00 sec 132 GBytes 18.9 Gbits/sec receiver
With scatter-gather IO:
root@u408d5cb7e82359d64def:~# iperf3 -c 172.16.0.1 -t 60
Connecting to host 172.16.0.1, port 5201
[ 4] local 172.16.0.2 port 42500 connected to 172.16.0.1 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 2.24 GBytes 19.3 Gbits/sec 0 942 KBytes
[ 4] 1.00-2.00 sec 2.12 GBytes 18.2 Gbits/sec 0 942 KBytes
[ 4] 2.00-3.00 sec 2.15 GBytes 18.4 Gbits/sec 0 942 KBytes
[ 4] 3.00-4.00 sec 2.16 GBytes 18.6 Gbits/sec 0 942 KBytes
[ 4] 4.00-5.00 sec 2.05 GBytes 17.6 Gbits/sec 0 942 KBytes
[ 4] 5.00-6.00 sec 2.11 GBytes 18.2 Gbits/sec 0 942 KBytes
[ 4] 6.00-7.00 sec 2.15 GBytes 18.4 Gbits/sec 0 942 KBytes
[ 4] 7.00-8.00 sec 2.15 GBytes 18.4 Gbits/sec 0 942 KBytes
[ 4] 8.00-9.00 sec 2.12 GBytes 18.2 Gbits/sec 0 942 KBytes
[ 4] 9.00-10.00 sec 2.13 GBytes 18.3 Gbits/sec 0 942 KBytes
[ 4] 10.00-11.00 sec 2.10 GBytes 18.0 Gbits/sec 0 942 KBytes
[ 4] 11.00-12.00 sec 2.14 GBytes 18.4 Gbits/sec 0 942 KBytes
[ 4] 12.00-13.00 sec 2.21 GBytes 19.0 Gbits/sec 0 942 KBytes
[ 4] 13.00-14.00 sec 2.13 GBytes 18.3 Gbits/sec 0 942 KBytes
[ 4] 14.00-15.00 sec 2.13 GBytes 18.3 Gbits/sec 0 942 KBytes
[ 4] 15.00-16.00 sec 2.19 GBytes 18.9 Gbits/sec 0 942 KBytes
[ 4] 16.00-17.00 sec 2.22 GBytes 19.1 Gbits/sec 0 942 KBytes
[ 4] 17.00-18.00 sec 2.17 GBytes 18.6 Gbits/sec 0 942 KBytes
[ 4] 18.00-19.00 sec 2.12 GBytes 18.2 Gbits/sec 0 942 KBytes
[ 4] 19.00-20.00 sec 2.09 GBytes 17.9 Gbits/sec 0 942 KBytes
[ 4] 20.00-21.00 sec 2.12 GBytes 18.2 Gbits/sec 0 942 KBytes
[ 4] 21.00-22.00 sec 2.17 GBytes 18.6 Gbits/sec 0 942 KBytes
[ 4] 22.00-23.00 sec 2.00 GBytes 17.2 Gbits/sec 0 942 KBytes
[ 4] 23.00-24.00 sec 1.88 GBytes 16.2 Gbits/sec 0 942 KBytes
[ 4] 24.00-25.00 sec 2.15 GBytes 18.5 Gbits/sec 0 942 KBytes
[ 4] 25.00-26.00 sec 2.08 GBytes 17.9 Gbits/sec 0 942 KBytes
[ 4] 26.00-27.00 sec 2.18 GBytes 18.7 Gbits/sec 0 942 KBytes
[ 4] 27.00-28.00 sec 2.06 GBytes 17.7 Gbits/sec 0 942 KBytes
[ 4] 28.00-29.00 sec 2.13 GBytes 18.2 Gbits/sec 0 942 KBytes
[ 4] 29.00-30.00 sec 2.16 GBytes 18.5 Gbits/sec 0 942 KBytes
[ 4] 30.00-31.00 sec 2.02 GBytes 17.4 Gbits/sec 0 942 KBytes
[ 4] 31.00-32.00 sec 2.13 GBytes 18.3 Gbits/sec 0 942 KBytes
[ 4] 32.00-33.00 sec 2.19 GBytes 18.8 Gbits/sec 0 942 KBytes
[ 4] 33.00-34.00 sec 2.09 GBytes 17.9 Gbits/sec 0 942 KBytes
[ 4] 34.00-35.00 sec 2.10 GBytes 18.0 Gbits/sec 0 942 KBytes
[ 4] 35.00-36.00 sec 2.08 GBytes 17.9 Gbits/sec 0 942 KBytes
[ 4] 36.00-37.00 sec 2.12 GBytes 18.2 Gbits/sec 0 942 KBytes
[ 4] 37.00-38.00 sec 2.17 GBytes 18.7 Gbits/sec 0 942 KBytes
[ 4] 38.00-39.00 sec 2.10 GBytes 18.0 Gbits/sec 0 942 KBytes
[ 4] 39.00-40.00 sec 2.12 GBytes 18.2 Gbits/sec 0 942 KBytes
[ 4] 40.00-41.00 sec 1.90 GBytes 16.3 Gbits/sec 0 942 KBytes
[ 4] 41.00-42.00 sec 2.16 GBytes 18.6 Gbits/sec 0 942 KBytes
[ 4] 42.00-43.00 sec 2.20 GBytes 18.9 Gbits/sec 0 942 KBytes
[ 4] 43.00-44.00 sec 2.17 GBytes 18.7 Gbits/sec 0 942 KBytes
[ 4] 44.00-45.00 sec 2.18 GBytes 18.7 Gbits/sec 0 942 KBytes
[ 4] 45.00-46.00 sec 2.21 GBytes 19.0 Gbits/sec 0 942 KBytes
[ 4] 46.00-47.00 sec 2.19 GBytes 18.8 Gbits/sec 0 942 KBytes
[ 4] 47.00-48.00 sec 2.17 GBytes 18.6 Gbits/sec 0 942 KBytes
[ 4] 48.00-49.00 sec 2.21 GBytes 19.0 Gbits/sec 0 942 KBytes
[ 4] 49.00-50.00 sec 2.18 GBytes 18.7 Gbits/sec 0 942 KBytes
[ 4] 50.00-51.00 sec 2.07 GBytes 17.8 Gbits/sec 0 942 KBytes
[ 4] 51.00-52.00 sec 2.18 GBytes 18.7 Gbits/sec 0 942 KBytes
[ 4] 52.00-53.00 sec 2.19 GBytes 18.8 Gbits/sec 0 942 KBytes
[ 4] 53.00-54.00 sec 2.03 GBytes 17.4 Gbits/sec 0 942 KBytes
[ 4] 54.00-55.00 sec 2.07 GBytes 17.8 Gbits/sec 0 942 KBytes
[ 4] 55.00-56.00 sec 2.09 GBytes 18.0 Gbits/sec 0 942 KBytes
[ 4] 56.00-57.00 sec 2.04 GBytes 17.5 Gbits/sec 0 942 KBytes
[ 4] 57.00-58.00 sec 2.07 GBytes 17.8 Gbits/sec 0 942 KBytes
[ 4] 58.00-59.00 sec 2.01 GBytes 17.3 Gbits/sec 0 942 KBytes
[ 4] 59.00-60.00 sec 2.03 GBytes 17.5 Gbits/sec 0 942 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.00 sec 127 GBytes 18.2 Gbits/sec 0 sender
[ 4] 0.00-60.00 sec 127 GBytes 18.2 Gbits/sec receiver
This being said, I would close this issue without any action. @firecracker-microvm/compute-capsule any opinion ?
This being said, I would close this issue without any action.
Looks like it. It'd be cool if you could keep that branch around or find a long-term place for the patches, maybe we come back to it if there is ever an actual need for higher net TX throughput.
In the meantime I vote we close the issue.
I tried a small optimization where only the MAC is read individually from the frame. Here is the commit. These are the results:
root@u408d5cb7e82359d64def:~# iperf3 -c 172.16.0.1 -t 60
Connecting to host 172.16.0.1, port 5201
[ 4] local 172.16.0.2 port 44940 connected to 172.16.0.1 port 5201
[ 35.582274] random: crng init done
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 2.04 GBytes 17.5 Gbits/sec 0 929 KBytes
[ 4] 1.00-2.00 sec 2.32 GBytes 19.9 Gbits/sec 0 929 KBytes
[ 4] 2.00-3.00 sec 2.32 GBytes 19.9 Gbits/sec 0 929 KBytes
[ 4] 3.00-4.00 sec 2.15 GBytes 18.5 Gbits/sec 0 929 KBytes
[ 4] 4.00-5.00 sec 2.27 GBytes 19.5 Gbits/sec 0 929 KBytes
[ 4] 5.00-6.00 sec 2.36 GBytes 20.3 Gbits/sec 0 929 KBytes
[ 4] 6.00-7.00 sec 2.31 GBytes 19.9 Gbits/sec 0 929 KBytes
[ 4] 7.00-8.00 sec 2.26 GBytes 19.4 Gbits/sec 0 929 KBytes
[ 4] 8.00-9.00 sec 2.27 GBytes 19.5 Gbits/sec 0 929 KBytes
[ 4] 9.00-10.00 sec 2.31 GBytes 19.8 Gbits/sec 0 929 KBytes
[ 4] 10.00-11.00 sec 2.30 GBytes 19.7 Gbits/sec 0 929 KBytes
[ 4] 11.00-12.00 sec 2.29 GBytes 19.7 Gbits/sec 0 929 KBytes
[ 4] 12.00-13.00 sec 2.32 GBytes 20.0 Gbits/sec 0 929 KBytes
[ 4] 13.00-14.00 sec 2.35 GBytes 20.2 Gbits/sec 0 929 KBytes
[ 4] 14.00-15.00 sec 2.27 GBytes 19.5 Gbits/sec 0 929 KBytes
[ 4] 15.00-16.00 sec 2.29 GBytes 19.7 Gbits/sec 0 929 KBytes
[ 4] 16.00-17.00 sec 2.33 GBytes 20.1 Gbits/sec 0 929 KBytes
[ 4] 17.00-18.00 sec 2.33 GBytes 20.0 Gbits/sec 0 929 KBytes
[ 4] 18.00-19.00 sec 2.28 GBytes 19.6 Gbits/sec 0 929 KBytes
[ 4] 19.00-20.00 sec 2.28 GBytes 19.6 Gbits/sec 0 929 KBytes
[ 4] 20.00-21.00 sec 2.38 GBytes 20.4 Gbits/sec 0 929 KBytes
[ 4] 21.00-22.00 sec 2.29 GBytes 19.7 Gbits/sec 0 929 KBytes
[ 4] 22.00-23.00 sec 2.27 GBytes 19.5 Gbits/sec 0 929 KBytes
[ 4] 23.00-24.00 sec 2.25 GBytes 19.3 Gbits/sec 0 929 KBytes
[ 4] 24.00-25.00 sec 2.36 GBytes 20.3 Gbits/sec 0 929 KBytes
[ 4] 25.00-26.00 sec 2.31 GBytes 19.9 Gbits/sec 0 929 KBytes
[ 4] 26.00-27.00 sec 2.25 GBytes 19.3 Gbits/sec 0 929 KBytes
[ 4] 27.00-28.00 sec 2.33 GBytes 20.0 Gbits/sec 0 929 KBytes
[ 4] 28.00-29.00 sec 2.29 GBytes 19.7 Gbits/sec 0 929 KBytes
[ 4] 29.00-30.00 sec 2.27 GBytes 19.5 Gbits/sec 0 929 KBytes
[ 4] 30.00-31.00 sec 2.30 GBytes 19.8 Gbits/sec 0 929 KBytes
[ 4] 31.00-32.00 sec 2.31 GBytes 19.9 Gbits/sec 0 929 KBytes
[ 4] 32.00-33.00 sec 2.29 GBytes 19.7 Gbits/sec 0 929 KBytes
[ 4] 33.00-34.00 sec 2.19 GBytes 18.9 Gbits/sec 0 929 KBytes
[ 4] 34.00-35.00 sec 2.31 GBytes 19.8 Gbits/sec 0 929 KBytes
[ 4] 35.00-36.00 sec 2.36 GBytes 20.3 Gbits/sec 0 929 KBytes
[ 4] 36.00-37.00 sec 2.27 GBytes 19.5 Gbits/sec 0 929 KBytes
[ 4] 37.00-38.00 sec 2.13 GBytes 18.3 Gbits/sec 0 929 KBytes
[ 4] 38.00-39.00 sec 2.28 GBytes 19.6 Gbits/sec 0 929 KBytes
[ 4] 39.00-40.00 sec 2.37 GBytes 20.4 Gbits/sec 0 929 KBytes
[ 4] 40.00-41.00 sec 2.33 GBytes 20.0 Gbits/sec 0 929 KBytes
[ 4] 41.00-42.00 sec 2.34 GBytes 20.1 Gbits/sec 0 929 KBytes
[ 4] 42.00-43.00 sec 2.37 GBytes 20.4 Gbits/sec 0 929 KBytes
[ 4] 43.00-44.00 sec 2.27 GBytes 19.5 Gbits/sec 0 929 KBytes
[ 4] 44.00-45.00 sec 2.35 GBytes 20.2 Gbits/sec 0 929 KBytes
[ 4] 45.00-46.00 sec 2.30 GBytes 19.8 Gbits/sec 0 929 KBytes
[ 4] 46.00-47.00 sec 2.36 GBytes 20.3 Gbits/sec 0 929 KBytes
[ 4] 47.00-48.00 sec 2.23 GBytes 19.2 Gbits/sec 0 929 KBytes
[ 4] 48.00-49.00 sec 2.25 GBytes 19.3 Gbits/sec 0 929 KBytes
[ 4] 49.00-50.00 sec 2.25 GBytes 19.3 Gbits/sec 0 929 KBytes
[ 4] 50.00-51.00 sec 2.33 GBytes 20.0 Gbits/sec 0 929 KBytes
[ 4] 51.00-52.00 sec 2.29 GBytes 19.7 Gbits/sec 0 929 KBytes
[ 4] 52.00-53.00 sec 2.25 GBytes 19.4 Gbits/sec 0 929 KBytes
[ 4] 53.00-54.00 sec 2.26 GBytes 19.4 Gbits/sec 0 929 KBytes
[ 4] 54.00-55.00 sec 2.32 GBytes 19.9 Gbits/sec 0 929 KBytes
[ 4] 55.00-56.00 sec 2.21 GBytes 19.0 Gbits/sec 0 929 KBytes
[ 4] 56.00-57.00 sec 2.21 GBytes 19.0 Gbits/sec 0 929 KBytes
[ 4] 57.00-58.00 sec 2.24 GBytes 19.2 Gbits/sec 0 929 KBytes
[ 4] 58.00-59.00 sec 2.06 GBytes 17.7 Gbits/sec 0 929 KBytes
[ 4] 59.00-60.00 sec 2.19 GBytes 18.8 Gbits/sec 0 929 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.00 sec 137 GBytes 19.6 Gbits/sec 0 sender
[ 4] 0.00-60.00 sec 137 GBytes 19.6 Gbits/sec receiver
There is a 0.7 Gbps improvement compared to the old version. So maybe it would be worth implementing this solution.