[WIP] Chunk distribution strategies

[franzpoeschel]

trafficstars

~~Based on topic-available-chunks, see #802.~~ Based on #1043

Adds strategies for guiding applications to efficient parallel chunk loading patterns. Idea: PR #802 lets reading applications inquire the chunks that are available for loading in the backend. This PR adds a function chunk_assignment::assignChunks, taking such a ChunkTable as well as a strategy as input and produces a further ChunkTable that may be used as a load pattern.

Implemented strategies:

• Round Robin: Assign chunks to reading processes in turn.
• by hostname: Assign chunks to processes on the same host. Parameterized by the interior distribution strategy.
• by cuboid slice: Assign chunks according to a slicing into n_ranks hyperslabs of the data space.
• by binpacking: Assign chunks in an approximatively balanced manner via an approximation algorithm for the NP complete binpacking problem.

Also, add an attribute to the global Series object (currently called rankMetaInfo, I'm sure we'll change this) that can be used to write string-formatted meta information for the single writing processes and assign meaning to them (e.g. hostnames). Used by the hostname-based strategy.

TODO:

• [x] Python bindings
• [ ] Testing, mostly done manually so far
• [ ] Cleanup
• [ ] Documentation
• [ ] This PR adds many MPI-based methods to the Series class. It would probably be good to first separate MPI and non-MPI headers, and then to add this functionality to the MPI headers.

[lgtm-com[bot]]

This pull request introduces 1 alert when merging 96841f9dbf402e940424ae63bc34b7612884ee0e into b2664b7b6b73552a69973217db2cd2f459dd354d - view on LGTM.com

new alerts:

• 1 for Catching by value

[lgtm-com[bot]]

This pull request introduces 1 alert when merging 6da1db3690f25dea6e39959fc4cf75a61e1ada8b into a6287fc1ffc121412839bbde5647794cef74a25a - view on LGTM.com

new alerts:

• 1 for Catching by value

[lgtm-com[bot]]

This pull request introduces 1 alert when merging 43b1e31f096978332d31495e7a217d185106bf0b into f6b0054841518aa9addc91d75847a0c75425b7dd - view on LGTM.com

new alerts:

• 1 for Catching by value

[lgtm-com[bot]]

This pull request introduces 1 alert when merging 5f2b347cb86612b936a8a52641c80184882a0da0 into f6b0054841518aa9addc91d75847a0c75425b7dd - view on LGTM.com

new alerts:

• 1 for Catching by value

[lgtm-com[bot]]

This pull request introduces 1 alert when merging bb7fa45aa65cd7c0756b83a1fffdd1698eba6928 into 9b349eb31bea7218d41d404cd5857896d7e2b557 - view on LGTM.com

new alerts:

• 1 for Catching by value

[franzpoeschel]

@ax3l ping

[lgtm-com[bot]]

This pull request introduces 1 alert when merging ebea6cf523463a6f4286c86571f0f8f81e43e8d3 into 86a15bafb6f8074e973483e6effb3e7e31c7ccb9 - view on LGTM.com

new alerts:

• 1 for Unused local variable

[franzpoeschel]

I've added Python bindings now, including usage of one distribution strategy in openpmd-pipe as an integrated usage example.

[franzpoeschel]

One issue of this approach is that the attribute rankMetaInfo is a vector of as many strings as there are parallel writers which I suspect is part of the reason why we saw scaling issues in some benchmarks. Instead, a parallel dataset should preferrably be written.

EDIT: I've transformed this to a parallel dataset now. Needs some more checking for char portability. Also, we should probably buffer these values on reading, since the table can only be read in steps in which it was written. In short: Make this less like an attribute that can be set and inquired, and rather more automatic during Series construction.

[lgtm-com[bot]]

This pull request introduces 1 alert when merging 646250c820348f8be84d0bbdaf3b6b539aee9b2a into 36506551e46b062badfbfef0db827eeeae9983dd - view on LGTM.com

new alerts:

• 1 for Multiplication result converted to larger type

---

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out our post on the GitHub blog.

[lgtm-com[bot]]

This pull request introduces 1 alert when merging ba5b8f90365e724a7c5776845d712eeca9d660f4 into 36506551e46b062badfbfef0db827eeeae9983dd - view on LGTM.com

new alerts:

• 1 for Multiplication result converted to larger type

---

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out our post on the GitHub blog.