micro-manager icon indicating copy to clipboard operation
micro-manager copied to clipboard

Parallelization fails on some machines with `MPI_ERR_TAG: invalid tag`

Open IshaanDesai opened this issue 5 months ago • 0 comments

The parallelization fails with the error:

  File "/home/ac137996/.local/lib/python3.11/site-packages/micro_manager/adaptivity/global_adaptivity.py", line 385, in _p2p_comm
    recv_reqs = self._p2p_comm(assoc_active_ids, micro_output)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ac137996/.local/lib/python3.11/site-packages/micro_manager/adaptivity/global_adaptivity.py", line 385, in _p2p_comm
    req = self._comm.isend(data[local_id], dest=send_rank, tag=tag)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/mpi4py/MPI.src/Comm.pyx", line 2006, in mpi4py.MPI.Comm.isend
    req = self._comm.irecv(bufsize, source=recv_rank, tag=tag)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/mpi4py/MPI.src/msgpickle.pxi", line 425, in mpi4py.MPI.PyMPI_isend
mpi4py.MPI.Exception: MPI_ERR_TAG: invalid tag

on some machines. @mathiskelm found that the size of the tags is the problem.

IshaanDesai avatar Sep 03 '24 08:09 IshaanDesai