open-mpi
coll tuned alltoall algorithm ignored after initialization
Thank you for taking the time to submit an issue!
Background information
What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)
v5.0.3
Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)
git clone
If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.
Please describe the system on which you are running
- Operating system/version: Ubuntu 22.04
- Computer hardware: Intel Xeon
- Network type: Infiband
Details of the problem
I would like to programmatically select the alltoall algorithm coll_tuned_alltoall_algorithm using the MPI_T interface based on communicator size and message size (amount of data sent by each rank). The problem is that coll_tuned_alltoall_algorithm is captured during communicator creation, it is cached and reused, and thus not changeable there after. This means that it is impossible to change the algorithm based on message size, which is not known until much later when MPI_Alltoall is called.
the value of the mca variable is captured and cached here: https://github.com/open-mpi/ompi/blob/d257a01ca3d298c3c7f8ff482f57120db6fcc098/ompi/mca/coll/tuned/coll_tuned_module.c#L240
the cached value is used here: https://github.com/open-mpi/ompi/blob/d257a01ca3d298c3c7f8ff482f57120db6fcc098/ompi/mca/coll/tuned/coll_tuned_decision_dynamic.c#L121
this caching prevents one from selecting the alltoall algorithm based on message size, which is an essential parameter in choosing the best algorithm. the mca variable could be queried during the dynamic decision process instead of caching.
