Doc: MPI Sync?

[ax3l]

Open brain storming if we want to document which calls are collective and synchronizing.

Wording loosely after the MPI standard, section 2.4: https://www.mpi-forum.org/docs/mpi-3.1/mpi31-report.pdf

[franzpoeschel]

On collective operations:

A procedure is collective if all processes in a process group need to invoke the procedure. A collective call may or may not be synchronizing. Collective calls over the same communicator must be executed in the same order by all members of the process group.

On the distinction between collective and synchronizing:

Collective operations can (but are not required to) complete as soon as the caller’s participation in the collective communication is finished. A blocking operation is complete as soon as the call returns. A nonblocking (immediate) call requires a separate completion call (cf. Section 3.7). The completion of a collective operation indicates that the caller is free to modify locations in the communication buffer. It does not indicate that other processes in the group have completed or even started the operation (unless otherwise implied by the description of the operation). Thus, a collective communication operation may, or may not, have the effect of synchronizing all calling processes. This statement excludes, of course, the barrier operation.

Our usage diverges from both terms:

• collective: Those operations which we declare as collective, but not synchronizing, may be executed in divergent orders on different MPI ranks. The above definition requires that the orders should be the same. (This is somewhat fine, we just document stricter requirements than needed)
• synchronizing is a guarantee that the operation implements MPI_Barrier() semantics. However, when series.flush() returns, we do not actually guarantee to our users that all ranks have reached the flush operation. We should avoid this term.

Suggestion:

• Use the term collective for operations that must be executed by all MPI ranks in the same order. This is in agreement with the MPI terminology, and with it we do not make any guarantee that we cannot hold. This PR currently calls these operations synchronizing.
• MPI does not have a notion for "operation that is enqueued now and flushed later". These are similar to nonblocking operations, but not the same: Nonblocking operations are communication operations. What we want to document for operations such as setAttribute() are requirements about coherent data declaration. I suggest to use a term outside MPI terminology for that: coherent operations.

[ax3l]

Wouldn't "non-blocking collective" be the right description for our .flush()? (like MPI_IBarrier)

I think we just do not have as MPI has a "corresponding completion operation" (test/wait). (We kind of would need one generally, e.g., to ensure a checkpoint is done. So far I have closed, in doubt, to be sure. Of course, there is always the additional challenge that a consistent MPI view does not correspond to a consistent PFS view.)

[ax3l]

coherent data declaration

Or maybe even simpler: consistent data declaration?