Possible integration with OpenMP runtime

[msimberg]

There are many levels of potential integration with the OpenMP runtime to avoid contention between pika's and OpenMP's runtime. The most simple would be to replace the loop spawning std::threads for use by the scheduler (https://github.com/pika-org/pika/blob/7ce98e549c6d4faf77795d5fc85b802cde2fcc8e/libs/pika/thread_pools/include/pika/thread_pools/scheduled_thread_pool_impl.hpp#L312-L334) by an OpenMP parallel region with one scheduling loop spawned per OpenMP worker thread.

The semantics of this mode would be slightly different compared to the current setup in that the main thread participates in the the "thread pool" (does it have to? what about nowait?). Things like pika::start would no longer be possible (it would not return until the parallel region has finished) but pika::init would be the only option. This would also mean no suspend/resume but that would instead be somewhat naturally be handled by simply leaving/entering the OpenMP parallel region.

Another difficulty might be handling multiple thread pools (e.g. a separate MPI pool). One solution could be to simply not treat the MPI pool as a thread pool handled by the resource partitioner, but instead separately spawn a thread for it. Another solution would be to aggregate the creation of thread pools into a central place so that a single OpenMP parallel region can be started (currently each thread pool starts its own threads separately).

What the above integration does not provide is interoperability with e.g. OpenMP tasks. pika would be constantly running one "task" per OpenMP worker thread in this mode.

What the above integration does provide is less drama surrounding "multiple runtimes".