Simplify custom thread pool creation

[msimberg]

Currently creating a new thread pool with a custom scheduling policy is easy:

rp.create_thread_pool("default", hpx::resource::scheduling_policy::local_priority_fifo)

but any other customization requires using the callback interface. For example to add a single scheduler mode flag:

rp.create_thread_pool("default",
    [](hpx::threads::policies::callback_notifier& notifier,
        std::size_t num_threads, std::size_t thread_offset,
        std::size_t pool_index, std::string const& pool_name)
    -> std::unique_ptr<hpx::threads::detail::thread_pool_base>
    {
        typename local_priority_queue_scheduler::init_parameter_type init(num_threads);
        std::unique_ptr<local_priority_queue_scheduler> scheduler(new local_priority_queue_scheduler(init));

        auto mode = hpx::threads::policies::scheduler_mode(
            hpx::threads::policies::do_background_work |
            hpx::threads::policies::reduce_thread_priority |
            hpx::threads::policies::delay_exit |
            hpx::threads::policies::enable_elasticity);

        std::unique_ptr<hpx::threads::detail::thread_pool_base> pool(
            new hpx::threads::detail::scheduled_thread_pool<local_priority_queue_scheduler>(
                std::move(scheduler), notifier, pool_index, pool_name, mode,
                thread_offset));

        return pool;
    });

While flexible, the callback interface is unnecessarily complicated for most use cases.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[biddisco]

I would like to keep this issue alive. I will (slowly) work on it as part of my scheduler cleanup.