MPI tags not general enough

[s-mayani]

The tags found in src/Communicate/Tags.h for MPI communication start overlapping after a certain run size (i.e. number of MPI processes used), which causes correctness issues in code which relies on communication and uses these tags. For example, this issue became visible when running the test/solver/TestGaussian.cpp on more than 512 nodes on Perlmutter, each having 4 GPUs.

This is now temporarily fixed by increasing the absolute distance between the tags which may be used at the same time, by commit b27fa15ed95a322873500d70a57e5df58e32a04f.

However, this is still an issue that we will run into for bigger runs which may reach this overlap limit, therefore, a permanent solution for it should be found.

[s-mayani]

Digging more into the issue, the problem is not the MPI tags, but rather the buffer factory. It seems that the send and receive buffers are "stepping on each other" i.e. the same place in memory is being allocated to them. This is because the buffer IDs used to get a buffer from the buffer factory are overlapping in the send and receive operations. A solution would be to lock memory that is in use, i.e. not give out the memory associated to the same buffer ID if another rank has not finished using it.

More details are in this presentation. MPI_Tags_issue.pdf