Ian

Yeah, that is a good point about the extra dependency. I'll work on putting together a small PR for review that includes some of that info in the FAQ/Documentation.

As an update, I've also tried Open MPI 5.0rc6 with the same results. Freezing in the same spot etc.

@bosilca Thanks for taking a look! On my end I have to allow the test to run until 20-100K tests before it freezes. Each test takes a different amount of...

@bosilca Specifying `--mca btl self,tcp` has enabled me to trigger when running on a single node, thankfully in a fairly short period of time as well(when running on a large...

Oops, my bad on the typo. I tried again with the correct function params and I'm getting the same result. I was able to attach ARM DDT to the process...

I tried mixing up the order of the `irec` vs `isend` and the corresponding `MPI_Wait` calls as well. Same deadlock issue except it changes what is stuck inside the buffers.

@bosilca Do you have any thoughts on next steps to debug this issue? Could the thread synchronization issue be causing issues? Maybe switching away from barriers? I can also cause...

@bosilca Still get the same issue with the update command as well. I'll need to re-compile OMPI in debug mode to check if it fails the assert as well like...

Thanks for all the help on this issue everyone! I'm wondering if you(@devreal) could describe your setup a little bit more? Using the below command I still see the freeze...

@devreal Interesting, I am not able to get the issue to go away even restricting the network card usage as you described. Are you able to provide some more details...