Replace `threadmanager::wait` by moving blocking MPI communication to separate `std::thread`

[msimberg]

Context in https://github.com/eth-cscs/DLA-Future/pull/423. Summary: that PR introduced a threadmanager::wait in certain locations to make sure that no work will be scheduled anymore before doing a blocking MPI call. We should keep waiting for work to be completed as much as possible, but we can relax the requirement by moving blocking MPI communication to dedicated std::threads. This should allow us to remove the threadmanager::wait and I find this a more permanent solution (following the principle of "never block pika worker threads").

[msimberg]

Before doing this I would like to have something like https://github.com/pika-org/pika/issues/167 in pika. This would allow doing it relatively cleanly with

pika::this_thread::sync_wait(
    execution::schedule(
        execution::std_thread_scheduler{}) |
        execution::then(blocking_mpi_functions_go_here));