btl_tcp_links broken?

[asylvest]

Background information

I'm trying to use the btl_tcp_links option which does not appear to be working

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

4.0.0

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

From the source tarball; vanilla configure / make with no special configure options

Please describe the system on which you are running

• Operating system/version: RHEL 7
• Computer hardware: AWS C5 instances (Intel)
• Network type: TCP

---

Details of the problem

According to the FAQ at https://www.open-mpi.org/faq/?category=tcp#tcp-multi-links, btl_tcp_links is supposed to be working in 4.0.0, but I'm not seeing any performance difference when using it.

I'm using 2 x c5.18xlarge instances in AWS which have 25 Gbps connectivity via TCP (note these are not the c5n instances which have EFA). Using iperf3 with the -P option to drive multiple ports, I achieve over 24 Gbps when communicating between two instances.

I have a trivial MPI benchmark which uses one rank per host and sends 10 GB from one rank to the other via MPI_Send() / MPI_Recv(). Running like this:

mpirun -N 1 --bind-to none --hostfile hosts.txt /path/to/my/application

I get around 9 Gbps.

My understanding is I can do the equivalent of iperf3's -P flag by using the --mca btl_tcp_links flag with OpenMPI. I attempted to do this via:

mpirun --mca btl_tcp_links 4 -N 1 --bind-to none --hostfile hosts.txt /path/to/my/application

However, I still get the same 9 Gbps; I tried changing the value to 2 or 3 which also made no difference.

Is there an additional flag or anything else I need to set? Is there a way I can run in a verbose mode to tell if multiple ports really are open? Or is something still not working properly in 4.0.0? I do have some use cases where I have a small number of hosts that need to transfer a large amount of data so I would benefit from getting this working.

Thanks for the help.