libvma
libvma copied to clipboard
Mellanox
Rust TcpStream clone cannot send/recv through libvma
Subject
I have a system setup where listening the UDP stream from one port and send/recv TCP from the other port of my NIC. All of the parts are working except the cloned TcpStream part. Using: https://doc.rust-lang.org/std/net/struct.TcpStream.html#method.try_clone
Issue type
- [x] Bug report
- [ ] Feature request
Configuration:
- Product version libvma-9.8.40
- OS - Redhat 9.2
- OFED MLNX_OFED_LINUX-23.07-0.5.1.2-rhel9.2-x86_64
- Hardware - Mellanox MT2894 Family [ConnectX-6 Lx]
Actual behavior:
I am using a separate thread to send and receive heartbeats so I pass the cloned TcpStream to that thread. The main TcpStream can send and receive packets but The cloned TcpStream cannot send and receive any packets, and TcpStream::write_all returns with Broken pipe error. When I don't use libvma and offload with OS, everything works as expected.
Expected behavior:
libvma is expected to work same as the OS to offload the traffic from the cloned TcpStream.
Steps to reproduce:
Server gist: https://gist.github.com/boranby/02464d37e188ba1fe6032afc79ba5d50 Client gist: https://gist.github.com/boranby/787343c64d6c5292369fd722977641ea
Need to update the IP addresses and route the Multicast address via port0 and route the TCP connection via port1.
You should see that login packet and login accepted packets reaches, but heartbeats don't reach.
Hello @boranby, I think in your application scenario you meet libvma limitations such as VMA does not work as server/client from single process and it has restriction in loopback flow usage.
Hi @igor-ivanov , thanks for your response.
I am running the client and server in two different machines. Does libvma consider cloning a TcpStream to read/write from two threads as server/client from single process?
do you mean that application uses thread0 for read and thread1 for write from the same process? Note: it should be ok from libvma. I noticed 127.0.0.1 ip in application sources you provided.
It is more like
- thread0 to read/write TCP
- thread1 to receive UDP multicast and write to TCP
I suspected that TCPStream clone has the problem. When I use only one thread to read/write TCP, it works.