micro-manager
micro-manager copied to clipboard
Parallelization fails on some machines with `MPI_ERR_TAG: invalid tag`
The parallelization fails with the error:
File "/home/ac137996/.local/lib/python3.11/site-packages/micro_manager/adaptivity/global_adaptivity.py", line 385, in _p2p_comm
recv_reqs = self._p2p_comm(assoc_active_ids, micro_output)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ac137996/.local/lib/python3.11/site-packages/micro_manager/adaptivity/global_adaptivity.py", line 385, in _p2p_comm
req = self._comm.isend(data[local_id], dest=send_rank, tag=tag)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "src/mpi4py/MPI.src/Comm.pyx", line 2006, in mpi4py.MPI.Comm.isend
req = self._comm.irecv(bufsize, source=recv_rank, tag=tag)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "src/mpi4py/MPI.src/msgpickle.pxi", line 425, in mpi4py.MPI.PyMPI_isend
mpi4py.MPI.Exception: MPI_ERR_TAG: invalid tag
on some machines. @mathiskelm found that the size of the tags is the problem.