devito icon indicating copy to clipboard operation
devito copied to clipboard

SparseFunctions + MPI + coordinates

Open FabioLuporini opened this issue 4 years ago • 2 comments

Assume rank0, out of 4 MPI ranks, has 8 items

a b c d e f g h

Now assume that, upon Operator application, the sparse points get distributed via _dist_scatter to the 4 ranks and the result is

a c  --- b d g --- e f --- h

Now, upon returning from Operator application, via _dist_gather, the sparse points get stored either as

a c b d g e f h

OR

a b e h c d f g

(to be double checked)

So the layout is different than it was upon data allocation/initialization.

Instead of making sure the layout is deterministically the same, currently the coordinates are rather reshuffled the same way the sparse points are, so that the mapping position -> value is correct

This is horrible. Possible approaches to improve this:

  • Fix the bug such that upon gather, the layout is as expected
  • Make SparseFunctions unordered collections (difficult because we do wanna use Data, I imagine...)

FabioLuporini avatar Sep 07 '19 07:09 FabioLuporini