CoreNeuron
CoreNeuron copied to clipboard
BlueBrain
Update MPI wrappers to be more flexible and handle C++/MPI types correctly
Currently MPI wrappers implemented in coreneuron/mpi/mpispike.cpp have quite some limitations:
- they were derived from old C style code
- for different argument types there are different functions i.e. code duplication (e.g. nrnmpi_dbl_allreduce_vec for double and long argument type)
- doesn't handle type like size_t correctly (see this)
- MPI Op type is passed as int 1, 2, 3 (quite unintuitive API)
- When #ranks < 2, not all functions avoid calling MPI functions
We should rewrite them to avoid repeated code and address above limitations. I assume @alkino will be interested to re-write this. @olupton and @ohm314 can provide input if these should be refactored/implemented in certain way.
P.S. the reason for avoiding calling MPI routines for #ranks < 2 is that MPI compiled library can be used in serial or parallel execution. In serial execution NEURON doesn't initialise MPI and libraries like HPE-MPT end-up with floating point exceptions.
Also, keep in mind that currently the passed operation for reductions is not the MPI_Opt code, but another user-defined opreation code that gets translated.
I don't know if this is relevant to CoreNEURON but one of the reasons for the c api to all this in NEURON is to allow dynamic loading of libmpi (if it exists). This is useful for binary distributions installed on machines where mpi may not be installed, and also be somewhat agnostic about which mpi is installed.
but one of the reasons for the c api to all this in NEURON is to allow dynamic loading of libmpi (if it exists).
@nrnhines : but even if we implement MPI wrappers in C++ with template parameters (to be type agnostic), we can still compile libnrnmpi.so and that can be dynamically loaded?
I still have a lot to learn about modern c++. If that is the case (it wasn't true in the 1990's, and we need to replace all the elaborate macros that were needed because templates didn't work) then go for it.