hp3d
hp3d copied to clipboard
Blocking directives in mesh migration
Mesh migration can be rather slow on large distributed meshes because each node is sent and received individually with blocking directives. I don't have a large file at the moment but on a moderate-size acoustics problem run on 64 Frontera nodes the timing was:
migration time: 4.14451 seconds
This becomes increasingly expensive on large meshes.
Indirect communication and/or batching may be needed.
As an additional comment: at the moment, a lot of node data (except for middle nodes) are sent/received redundantly; there are various ways of avoiding this which should help with the performance of the data migration.