MPI calls hang with random node allocation

[AKKamath]

MPI calls hang with random node allocation

Assuming a topology has been set up (I used the default dragonfly), the following config causes the system to hang:

    ep = EmberMPIJob(1, nnodes)
    ep.network_interface = networkif
    
    # Add motifs to the simulation
    ep.addMotif("Init")
    ep.addMotif("Allgather")
    ep.addMotif("Fini")
    
    system = System()
    system.setTopology(topo)
    system.allocateNodes(ep,"random")
    system.build()

I've isolated a few reasons why this is happening.

When the GroupWorld is initialized it is given a map of rank IDs to node IDs, which in the above case is randomly allocated: group->initMapping( &(*m_sreg.begin()), m_netMapSize, m_virtNic->getNumCores() );

So if we have 4 randomly selected nodes in a network of 100 nodes, we may possibly get the following: Rank 0 -> Node 22 Rank 1 -> Node 56 Rank 2 -> Node 7 Rank 3 -> Node 10

When issuing send operations, the allgather operator converts the location to send using the MPI communicator provided, as seen below:

    proto()->send( addr, 0, **m_info->getGroup(m_event->group)->getMapping( m_dest[0] )**, genTag(), m_smallCollectiveVN );

Assume, we're sending to rank 1 from rank 0, the send operation will be called with destination = 56.

Internally, the send operation assumes a rank ID is sent, and so again converts from rank to node ID: nid_t nid = calcNid( req, req->getDestRank() );

This double conversion is causing the hangs, as send operations are being sent to the wrong node IDs, and nodes end up waiting to receive from other node IDs. I'm not sure what a clean solution for this is.

[feldergast]

Can you give me your whole configuration? We've run a lot of simulations using random allocations with no issues, so I need to see if there's something subtle going on in the configuration.

[AKKamath]

Sure. Github doesn't allow .py files so I've attached it as a .txt here: allgather.txt