Lost status is_ready from future

[m-diers]

Expected Behavior

The simple scheduling example is to go through to the end and distribute the work equally among the localities.

Actual Behavior

It is served after some time only the first locality. The future<> get no is_ready state although they are processed. Afterwards the application remains completely in the distribution loop, because all future<> get no is_ready.

We have had this behavior for a very long time, but have never been able to properly reproduce it to identify the dependencies. The example has now been parameterized as far as possible to make the problem as reproducible as possible ;-)

Steps to Reproduce the Problem

• example scheduling code https://gist.github.com/m-diers/f1cdec7cf48715e0ab80a0034e9ccaf8
• compile full release
• hardware AMD Ryzen 9 5950X (16 core, enabled hyperthreading)
• run

srun -n 16 \
     -N 1 \
     -w "MYNODE" \
     ./scheduletest \
     --hpx:ini=hpx.parcel.mpi.enable!=1 \
     --hpx:ini=hpx.parcel.tcp.enable!=0

Specifications

• HPX Version: master (well reproducible)
• HPX Version: 1.7.1 (badly reproducible)
• Compile optimizations: release
• Platform (compiler, OS): clang 12, OpenMPI 4.0.3, Ubuntu 20.04.03, Docker/Singularity
• It is hardware dependent how well it can be reproduced (timing)
• Process per core, as this is the best way to reproduce the behavior in my case
• Parcelport mpi: well reproducible
• Parcelport tcp: badly reproducible

[hkaiser]

@m-diers I have asked @hcq9102 to try to reproduce your issue. She will be in contact, if needed.

[m-diers]

Is there any other help needed to reproduce? I can also reproduce it on other nodes with srun -N 1 -n ${NumberOfPhysicalCores} With mpirun the same behavior should occur.

[hcq9102]

Is there any other help needed to reproduce? I can also reproduce it on other nodes with srun -N 1 -n ${NumberOfPhysicalCores} With mpirun the same behavior should occur.

Hey @m-diers, apologize for the late response. I was busy with the course last week, I will start this tmr and back to you ASAP.

[hcq9102]

Hi @m-diers, I was trying to reproduce the problem on our Rostam cluster. I didn't even get the executable file after these: https://gist.github.com/hcq9102/84b5a0e639084323de15414aa9668be0 could you please help check that? THANKS.

[m-diers]

Hi @m-diers, I was trying to reproduce the problem on our Rostam cluster. I didn't even get the executable file after these: https://gist.github.com/hcq9102/84b5a0e639084323de15414aa9668be0 could you please help check that? THANKS.

I suspect that creating the component with add_hpx_component is redundant. As a result, the symbols of scheduletest.cpp are duplicated.

[hcq9102]

Hi @m-diers, I was trying to reproduce the problem on our Rostam cluster. I didn't even get the executable file after these: https://gist.github.com/hcq9102/84b5a0e639084323de15414aa9668be0 could you please help check that? THANKS.

I suspect that creating the component with add_hpx_component is redundant. As a result, the symbols of scheduletest.cpp are duplicated.

At the very beginning, I didn't have add_hpx_component in CMakeList.txt, but it complained scheduletest.cpp:(.text+0x1bd8): undefined reference to hpx::cout , you may check and see here: issue. And The original CMakeList.txt file as listed at comment(Is this the one you are using?)

I also tried to delete add_executable(scheduletest scheduletest.cpp) when I added add_hpx_component(), but got Cmake complain issue. so I keep both of them here.

[m-diers]

Updated the CmakeList.txt https://gist.github.com/hcq9102/84b5a0e639084323de15414aa9668be0?permalink_comment_id=4093908#gistcomment-4093908

[hcq9102]

Updated the CmakeList.txt https://gist.github.com/hcq9102/84b5a0e639084323de15414aa9668be0?permalink_comment_id=4093908#gistcomment-4093908

Okay, Thx.

[hcq9102]

Hi @m-diers We are able to create the file. But we got some issues on our cluster when run the file with srun -n 16 -N 1 -p medusa ./ScheduleTest --hpx:ini=hpx.parcel.mpi.enable!=1 --hpx:ini=hpx.parcel.tcp.enable!=0 , and cannot reproduce the problem for now. We will continue working on this and update you once we figure out what's going on. Thanks.

[m-diers]

According to Running-HPX-on-Rostam, the medusa nodes have 40 physical cores. Then it should also be started with -n 40 -N 1. An interesting test could also be on kamand with -n 128 -N 1.

[hcq9102]

update: srun -n 40 -N 1 -p medusa ./ScheduleTest --hpx:ini=hpx.parcel.mpi.enable!=1 --hpx:ini=hpx.parcel.tcp.enable!=0 on rostam; Same issue as before:

[medusa00:118035] *** An error occurred in MPI_Init_thread
[medusa00:118035] *** reported by process [919928832,24]
[medusa00:118035] *** on a NULL communicator
[medusa00:118035] *** Unknown error
[medusa00:118035] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[medusa00:118035] ***    and potentially your MPI job)
[medusa00:118038] *** An error occurred in MPI_Init_thread
[medusa00:118038] *** reported by process [919928832,27]
[medusa00:118038] *** on a NULL communicator
[medusa00:118038] *** Unknown error
[medusa00:118038] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[medusa00:118038] ***    and potentially your MPI job)
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  PML add procs failed
  --> Returned "Not found" (-13) instead of "Success" (0)
--------------------------------------------------------------------------
[medusa00:118039] *** An error occurred in MPI_Init_thread
[medusa00:118039] *** reported by process [919928832,28]
[medusa00:118039] *** on a NULL communicator
[medusa00:118039] *** Unknown error
[medusa00:118039] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[medusa00:118039] ***    and potentially your MPI job)
srun: error: medusa00: tasks 0-39: Killed

[hkaiser]

update: srun -n 40 -N 1 -p medusa ./ScheduleTest --hpx:ini=hpx.parcel.mpi.enable!=1 --hpx:ini=hpx.parcel.tcp.enable!=0 on rostam; Same issue as before:

This very much looks to be unrelated...

[m-diers]

This very much looks to be unrelated...

Unfortunately I can not help with the cluster/slurm/mpi configuration...

[John98Zakaria]

I ran it locally on my machine using ( humble i5-4460) I didn't get any errors

My HPX Config

Static configuration:

Core library: HPX_AGAS_LOCAL_CACHE_SIZE=4096 HPX_HAVE_MALLOC=jemalloc HPX_PARCEL_MAX_CONNECTIONS=512 HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4 HPX_PREFIX (configured)= HPX_PREFIX=/home/jn98zk/cpp_stuff/hpx-1.7.1/installation

HPX_FILESYSTEM_WITH_BOOST_FILESYSTEM_COMPATIBILITY=OFF HPX_ITERATOR_SUPPORT_WITH_BOOST_ITERATOR_TRAVERSAL_TAG_COMPATIBILITY=OFF HPX_WITH_AGAS_DUMP_REFCNT_ENTRIES=OFF HPX_WITH_APEX=OFF HPX_WITH_ASYNC_MPI=OFF HPX_WITH_ATTACH_DEBUGGER_ON_TEST_FAILURE=OFF HPX_WITH_AUTOMATIC_SERIALIZATION_REGISTRATION=ON HPX_WITH_COROUTINE_COUNTERS=OFF HPX_WITH_DISTRIBUTED_RUNTIME=ON HPX_WITH_DYNAMIC_HPX_MAIN=ON HPX_WITH_GOOGLE_PERFTOOLS=OFF HPX_WITH_IO_COUNTERS=ON HPX_WITH_IO_POOL=ON HPX_WITH_ITTNOTIFY=OFF HPX_WITH_LOGGING=ON HPX_WITH_NETWORKING=ON HPX_WITH_PAPI=OFF HPX_WITH_PARALLEL_TESTS_BIND_NONE=OFF HPX_WITH_PARCELPORT_ACTION_COUNTERS=OFF HPX_WITH_PARCELPORT_LIBFABRIC=OFF HPX_WITH_PARCELPORT_MPI=ON HPX_WITH_PARCELPORT_MPI_MULTITHREADED=ON HPX_WITH_PARCELPORT_TCP=ON HPX_WITH_PARCEL_COALESCING=ON HPX_WITH_PARCEL_PROFILING=OFF HPX_WITH_SANITIZERS=OFF HPX_WITH_SCHEDULER_LOCAL_STORAGE=OFF HPX_WITH_SPINLOCK_DEADLOCK_DETECTION=ON HPX_WITH_STACKTRACES=ON HPX_WITH_STACKTRACES_DEMANGLE_SYMBOLS=ON HPX_WITH_STACKTRACES_STATIC_SYMBOLS=OFF HPX_WITH_TESTS_DEBUG_LOG=OFF HPX_WITH_THREAD_BACKTRACE_ON_SUSPENSION=OFF HPX_WITH_THREAD_CREATION_AND_CLEANUP_RATES=OFF HPX_WITH_THREAD_CUMULATIVE_COUNTS=ON HPX_WITH_THREAD_DEBUG_INFO=OFF HPX_WITH_THREAD_DESCRIPTION_FULL=OFF HPX_WITH_THREAD_GUARD_PAGE=ON HPX_WITH_THREAD_IDLE_RATES=OFF HPX_WITH_THREAD_LOCAL_STORAGE=OFF HPX_WITH_THREAD_MANAGER_IDLE_BACKOFF=ON HPX_WITH_THREAD_QUEUE_WAITTIME=OFF HPX_WITH_THREAD_STACK_MMAP=ON HPX_WITH_THREAD_STEALING_COUNTS=OFF HPX_WITH_THREAD_TARGET_ADDRESS=OFF HPX_WITH_TIMER_POOL=ON HPX_WITH_TUPLE_RVALUE_SWAP=ON HPX_WITH_VALGRIND=OFF HPX_WITH_VERIFY_LOCKS=OFF HPX_WITH_VERIFY_LOCKS_BACKTRACE=OFF HPX_WITH_VERIFY_LOCKS_GLOBALLY=OFF

Module coroutines: HPX_COROUTINES_WITH_SWAP_CONTEXT_EMULATION=OFF

Module datastructures: HPX_DATASTRUCTURES_WITH_ADAPT_STD_TUPLE=ON

Module serialization: HPX_SERIALIZATION_WITH_ALL_TYPES_ARE_BITWISE_SERIALIZABLE=OFF HPX_SERIALIZATION_WITH_BOOST_TYPES=ON

Module topology: HPX_TOPOLOGY_WITH_ADDITIONAL_HWLOC_TESTING=OFF

Runtime configuration:

{mode}: console {scheduler}: local-priority-fifo {os-threads}: 4 {cores}: 4

[hkaiser]

I ran it locally on my machine using ( humble i5-4460) I didn't get any errors

My HPX Config

Yes, we tried to reproduce it several times on different machines as well... no success.

[m-diers]

Thanks, also for the config from @John98Zakaria

The following values, which may affect the results, are still different from mine:

HPX_HAVE_MALLOC=tcmalloc
HPX_WITH_APEX=OFF
HPX_WITH_SPINLOCK_DEADLOCK_DETECTION=OFF
HPX_WITH_THREAD_IDLE_RATES=On

Even with the changed parameters I still get the same behavior, even with ubuntu 22.04 and hpx 1.8.0-rc2. Is there a dockerfile that I can take as a basis for my test? Maybe I can get further with it.

[m-diers]

After some time I found some time again. The problem still exists, even with the current github master version. I have noticed that the performance has become a bit worse, which then also makes the problem a bit harder to reproduce. Therefore I went back to the version 1.7.1, because I can reproduce it better with the version.

Here is now a call stack of my problem (with 1.7.1), all threads are somehow stuck in a wait:

Call Stack

# 1
pthread_cond_wait (@pthread_cond_wait:143)
void asio::detail::posix_event::wait<asio::detail::conditionally_enabled_mutex::scoped_lock>(asio::detail::conditionally_enabled_mutex::scoped_lock&) (/usr/include/asio/detail/posix_event.hpp:118)
asio::detail::conditionally_enabled_event::wait(asio::detail::conditionally_enabled_mutex::scoped_lock&) (/usr/include/asio/detail/conditionally_enabled_event.hpp:96)
asio::detail::scheduler::do_run_one(asio::detail::conditionally_enabled_mutex::scoped_lock&, asio::detail::scheduler_thread_info&, std::error_code const&) (/usr/include/asio/detail/impl/scheduler.ipp:489)
asio::detail::scheduler::run(std::error_code&) (/usr/include/asio/detail/impl/scheduler.ipp:203)
asio::io_context::run() (/usr/include/asio/impl/io_context.ipp:62)
hpx::util::io_service_pool::thread_run(unsigned long, hpx::util::barrier*) (@hpx::util::io_service_pool::thread_run(unsigned long, hpx::util::barrier*):56)
hpx::runtime_distributed::wait() (@hpx::runtime_distributed::wait():113)
hpx::runtime_distributed::run(hpx::util::function<int (), false> const&) (@hpx::runtime_distributed::run(hpx::util::function<int (), false> const&):11)
hpx::detail::run(hpx::runtime&, hpx::util::function<int (hpx::program_options::variables_map&), false> const&, hpx::program_options::variables_map&, hpx::runtime_mode, hpx::util::unique_function<void (), false>, hpx::util::unique_function<void (), false>) (@hpx::detail::run(hpx::runtime&, hpx::util::function<int (hpx::program_options::variables_map&), false> const&, hpx::program_options::variables_map&, hpx::runtime_mode, hpx::util::unique_function<void (), false>, hpx::util::unique_function<void (), false>):95)
hpx::detail::run_or_start(bool, std::unique_ptr<hpx::runtime, std::default_delete<hpx::runtime> >, hpx::util::command_line_handling&, hpx::util::unique_function<void (), false>, hpx::util::unique_function<void (), false>) (@hpx::detail::run_or_start(bool, std::unique_ptr<hpx::runtime, std::default_delete<hpx::runtime> >, hpx::util::command_line_handling&, hpx::util::unique_function<void (), false>, hpx::util::unique_function<void (), false>):37)
hpx::detail::run_or_start(hpx::util::function<int (hpx::program_options::variables_map&), false> const&, int, char**, hpx::init_params const&, bool) (@hpx::detail::run_or_start(hpx::util::function<int (hpx::program_options::variables_map&), false> const&, int, char**, hpx::init_params const&, bool):664)
hpx::detail::init_impl(hpx::util::function<int (hpx::program_options::variables_map&), false> const&, int, char**, hpx::init_params const&) (/opt/hpx-git/include/hpx/hpx_init_impl.hpp:77)
hpx::init(int, char**, hpx::init_params const&) (/opt/hpx-git/include/hpx/hpx_init_impl.hpp:117)
main (main.cpp:50)


# 2, 3
pthread_cond_wait (@pthread_cond_wait:143)
void asio::detail::posix_event::wait<asio::detail::conditionally_enabled_mutex::scoped_lock>(asio::detail::conditionally_enabled_mutex::scoped_lock&) (/usr/include/asio/detail/posix_event.hpp:118)
asio::detail::conditionally_enabled_event::wait(asio::detail::conditionally_enabled_mutex::scoped_lock&) (/usr/include/asio/detail/conditionally_enabled_event.hpp:96)
asio::detail::scheduler::do_run_one(asio::detail::conditionally_enabled_mutex::scoped_lock&, asio::detail::scheduler_thread_info&, std::error_code const&) (/usr/include/asio/detail/impl/scheduler.ipp:489)
asio::detail::scheduler::run(std::error_code&) (/usr/include/asio/detail/impl/scheduler.ipp:203)
asio::io_context::run() (/usr/include/asio/impl/io_context.ipp:62)
hpx::util::io_service_pool::thread_run(unsigned long, hpx::util::barrier*) (@hpx::util::io_service_pool::thread_run(unsigned long, hpx::util::barrier*):56)


# 4, 5
epoll_wait (@epoll_wait:27)
asio::detail::epoll_reactor::run(long, asio::detail::op_queue<asio::detail::scheduler_operation>&) (/usr/include/asio/detail/impl/epoll_reactor.ipp:470)
asio::detail::scheduler::do_run_one(asio::detail::conditionally_enabled_mutex::scoped_lock&, asio::detail::scheduler_thread_info&, std::error_code const&) (/usr/include/asio/detail/impl/scheduler.ipp:464)
asio::detail::scheduler::run(std::error_code&) (/usr/include/asio/detail/impl/scheduler.ipp:203)
asio::io_context::run() (/usr/include/asio/impl/io_context.ipp:62)
hpx::util::io_service_pool::thread_run(unsigned long, hpx::util::barrier*) (@hpx::util::io_service_pool::thread_run(unsigned long, hpx::util::barrier*):56)


# 6, 10, 12, 13, 15
__atomic_load (@__atomic_load:89)
std::atomic<boost::lockfree::tagged_ptr_pair<boost::lockfree::deque_node<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*>, boost::lockfree::deque_node<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*> > >::load(std::memory_order) const volatile (/usr/include/c++/12/atomic:294)
boost::lockfree::deque_anchor<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*>::lrs() const volatile (@boost::lockfree::deque<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*, boost::lockfree::caching_freelist_t, hpx::util::aligned_allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*> >::pop_right(hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*&):13)
boost::lockfree::deque<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*, boost::lockfree::caching_freelist_t, hpx::util::aligned_allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*> >::pop_right(hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*&) (@boost::lockfree::deque<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*, boost::lockfree::caching_freelist_t, hpx::util::aligned_allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*> >::pop_right(hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*&):13)
hpx::threads::policies::lockfree_fifo_backend<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*>::pop(hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*&, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::add_new(long, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, std::unique_lock<std::mutex>&, bool):30)
hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::add_new(long, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, std::unique_lock<std::mutex>&, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::add_new(long, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, std::unique_lock<std::mutex>&, bool):27)
hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::add_new_always(unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, std::unique_lock<std::mutex>&, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(bool, unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, bool):140)
hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(bool, unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(bool, unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, bool):108)
hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(unsigned long, bool, long&, bool, unsigned long&) (@hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(unsigned long, bool, long&, bool, unsigned long&):74)
void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>&, hpx::threads::detail::scheduling_counters&, hpx::threads::detail::scheduling_callbacks&) (@void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>&, hpx::threads::detail::scheduling_counters&, hpx::threads::detail::scheduling_callbacks&):714)
hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>) (@hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>):424)
void std::__invoke_impl<void, void (hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::*)(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>), hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >*, unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier> >(std::__invoke_memfun_deref, void (hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::*&&)(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>), hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >*&&, unsigned long&&, unsigned long&&, std::shared_ptr<hpx::util::barrier>&&) (/usr/include/c++/12/bits/invoke.h:74)


# 7, 17
hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::cleanup_terminated_locked(bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(bool, unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, bool):189)
hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(bool, unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(bool, unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, bool):188)
hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(unsigned long, bool, long&, bool, unsigned long&) (@hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(unsigned long, bool, long&, bool, unsigned long&):74)
void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>&, hpx::threads::detail::scheduling_counters&, hpx::threads::detail::scheduling_callbacks&) (@void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>&, hpx::threads::detail::scheduling_counters&, hpx::threads::detail::scheduling_callbacks&):714)
hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>) (@hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>):424)
void std::__invoke_impl<void, void (hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::*)(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>), hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >*, unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier> >(std::__invoke_memfun_deref, void (hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::*&&)(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>), hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >*&&, unsigned long&&, unsigned long&&, std::shared_ptr<hpx::util::barrier>&&) (/usr/include/c++/12/bits/invoke.h:74)


# 8, 11, 18, 20
__atomic_load (@__atomic_load:89)
std::atomic<boost::lockfree::tagged_ptr_pair<boost::lockfree::deque_node<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*>, boost::lockfree::deque_node<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*> > >::load(std::memory_order) const volatile (/usr/include/c++/12/atomic:294)
boost::lockfree::deque_anchor<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*>::lrs() const volatile (@boost::lockfree::deque<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*, boost::lockfree::caching_freelist_t, hpx::util::aligned_allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*> >::pop_right(hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*&):13)
boost::lockfree::deque<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*, boost::lockfree::caching_freelist_t, hpx::util::aligned_allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*> >::pop_right(hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*&) (@boost::lockfree::deque<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*, boost::lockfree::caching_freelist_t, hpx::util::aligned_allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*> >::pop_right(hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*&):13)
hpx::threads::policies::lockfree_fifo_backend<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*>::pop(hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*&, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::add_new(long, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, std::unique_lock<std::mutex>&, bool):30)
hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::add_new(long, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, std::unique_lock<std::mutex>&, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::add_new(long, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, std::unique_lock<std::mutex>&, bool):27)
hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::add_new_always(unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, std::unique_lock<std::mutex>&, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(bool, unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, bool):140)
hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(bool, unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(bool, unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, bool):108)
hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(unsigned long, bool, long&, bool, unsigned long&) (@hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(unsigned long, bool, long&, bool, unsigned long&):87)
void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>&, hpx::threads::detail::scheduling_counters&, hpx::threads::detail::scheduling_callbacks&) (@void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>&, hpx::threads::detail::scheduling_counters&, hpx::threads::detail::scheduling_callbacks&):714)
hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>) (@hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>):424)
void std::__invoke_impl<void, void (hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::*)(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>), hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >*, unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier> >(std::__invoke_memfun_deref, void (hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::*&&)(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>), hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >*&&, unsigned long&&, unsigned long&&, std::shared_ptr<hpx::util::barrier>&&) (/usr/include/c++/12/bits/invoke.h:74)


# 9
hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::add_new(long, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, std::unique_lock<std::mutex>&, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::add_new(long, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, std::unique_lock<std::mutex>&, bool):22)
hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::add_new_always(unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, std::unique_lock<std::mutex>&, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(bool, unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, bool):140)
hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(bool, unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(bool, unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, bool):108)
hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(unsigned long, bool, long&, bool, unsigned long&) (@hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(unsigned long, bool, long&, bool, unsigned long&):74)
void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>&, hpx::threads::detail::scheduling_counters&, hpx::threads::detail::scheduling_callbacks&) (@void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>&, hpx::threads::detail::scheduling_counters&, hpx::threads::detail::scheduling_callbacks&):714)
hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>) (@hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>):424)
void std::__invoke_impl<void, void (hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::*)(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>), hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >*, unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier> >(std::__invoke_memfun_deref, void (hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::*&&)(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>), hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >*&&, unsigned long&&, unsigned long&&, std::shared_ptr<hpx::util::barrier>&&) (/usr/include/c++/12/bits/invoke.h:74)


# 14
hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::get_next_thread(unsigned long, bool, hpx::threads::thread_data*&, bool) (@hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::get_next_thread(unsigned long, bool, hpx::threads::thread_data*&, bool):67)
void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>&, hpx::threads::detail::scheduling_counters&, hpx::threads::detail::scheduling_callbacks&) (@void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>&, hpx::threads::detail::scheduling_counters&, hpx::threads::detail::scheduling_callbacks&):111)
hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>) (@hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>):424)
void std::__invoke_impl<void, void (hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::*)(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>), hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >*, unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier> >(std::__invoke_memfun_deref, void (hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::*&&)(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>), hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >*&&, unsigned long&&, unsigned long&&, std::shared_ptr<hpx::util::barrier>&&) (/usr/include/c++/12/bits/invoke.h:74)


# 16
pthread_cond_clockwait (@pthread_cond_clockwait:132)
std::__condvar::wait_until(std::mutex&, int, timespec&) (/usr/include/c++/12/bits/std_mutex.h:169)
std::cv_status std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) (/usr/include/c++/12/condition_variable:200)
std::cv_status std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) (/usr/include/c++/12/condition_variable:110)
std::cv_status std::condition_variable::wait_for<long, std::ratio<1l, 1000l> >(std::unique_lock<std::mutex>&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&) (/usr/include/c++/12/condition_variable:162)
hpx::threads::policies::scheduler_base::idle_callback(unsigned long) (@hpx::threads::policies::scheduler_base::idle_callback(unsigned long):30)
hpx::util::detail::basic_function<void (), false, false>::operator()() const (@void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>&, hpx::threads::detail::scheduling_counters&, hpx::threads::detail::scheduling_callbacks&):438)
void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>&, hpx::threads::detail::scheduling_counters&, hpx::threads::detail::scheduling_callbacks&) (@void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>&, hpx::threads::detail::scheduling_counters&, hpx::threads::detail::scheduling_callbacks&):436)
hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>) (@hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>):424)
void std::__invoke_impl<void, void (hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::*)(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>), hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >*, unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier> >(std::__invoke_memfun_deref, void (hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::*&&)(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>), hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >*&&, unsigned long&&, unsigned long&&, std::shared_ptr<hpx::util::barrier>&&) (/usr/include/c++/12/bits/invoke.h:74)


# 19
boost::lockfree::deque<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*, boost::lockfree::caching_freelist_t, hpx::util::aligned_allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*> >::pop_right(hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*&) (@boost::lockfree::deque<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*, boost::lockfree::caching_freelist_t, hpx::util::aligned_allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*> >::pop_right(hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*&):18)
hpx::threads::policies::lockfree_fifo_backend<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*>::pop(hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*&, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::add_new(long, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, std::unique_lock<std::mutex>&, bool):30)
hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::add_new(long, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, std::unique_lock<std::mutex>&, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::add_new(long, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, std::unique_lock<std::mutex>&, bool):27)
hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::add_new_always(unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, std::unique_lock<std::mutex>&, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(bool, unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, bool):140)
hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(bool, unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(bool, unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, bool):108)
hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(unsigned long, bool, long&, bool, unsigned long&) (@hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(unsigned long, bool, long&, bool, unsigned long&):74)
void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>&, hpx::threads::detail::scheduling_counters&, hpx::threads::detail::scheduling_callbacks&) (@void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>&, hpx::threads::detail::scheduling_counters&, hpx::threads::detail::scheduling_callbacks&):714)
hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>) (@hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>):424)
void std::__invoke_impl<void, void (hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::*)(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>), hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >*, unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier> >(std::__invoke_memfun_deref, void (hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::*&&)(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>), hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >*&&, unsigned long&&, unsigned long&&, std::shared_ptr<hpx::util::barrier>&&) (/usr/include/c++/12/bits/invoke.h:74)


# 21
boost::lockfree::deque<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*, boost::lockfree::caching_freelist_t, hpx::util::aligned_allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*> >::pop_right(hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*&) (@boost::lockfree::deque<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*, boost::lockfree::caching_freelist_t, hpx::util::aligned_allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*> >::pop_right(hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*&):18)
hpx::threads::policies::lockfree_fifo_backend<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*>::pop(hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::task_description*&, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::add_new(long, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, std::unique_lock<std::mutex>&, bool):30)
hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::add_new(long, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, std::unique_lock<std::mutex>&, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::add_new(long, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, std::unique_lock<std::mutex>&, bool):27)
hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::add_new_always(unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, std::unique_lock<std::mutex>&, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(bool, unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, bool):140)
hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(bool, unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, bool) (@hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(bool, unsigned long&, hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>*, bool):108)
hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(unsigned long, bool, long&, bool, unsigned long&) (@hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>::wait_or_add_new(unsigned long, bool, long&, bool, unsigned long&):87)
void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>&, hpx::threads::detail::scheduling_counters&, hpx::threads::detail::scheduling_callbacks&) (@void hpx::threads::detail::scheduling_loop<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >(unsigned long, hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>&, hpx::threads::detail::scheduling_counters&, hpx::threads::detail::scheduling_callbacks&):714)
hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>) (@hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>):424)
void std::__invoke_impl<void, void (hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::*)(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>), hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >*, unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier> >(std::__invoke_memfun_deref, void (hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::*&&)(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>), hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >*&&, unsigned long&&, unsigned long&&, std::shared_ptr<hpx::util::barrier>&&) (/usr/include/c++/12/bits/invoke.h:74)


# 22
pthread_cond_wait (@pthread_cond_wait:143)
hpx::components::server::runtime_support::wait() (@hpx::components::server::runtime_support::wait():72)
hpx::runtime_distributed::wait_helper(std::mutex&, std::condition_variable&, bool&) (@hpx::runtime_distributed::wait_helper(std::mutex&, std::condition_variable&, bool&):69)

With the current asio-1.22.2 it is the same behavior.

[m-diers]

I replaced the https://gist.github.com/m-diers/f1cdec7cf48715e0ab80a0034e9ccaf8#file-scheduletest-cpp-L73

hpx::wait_any( results );

with

std::this_thread::yield();

and it does not stop anymore. ;-)

[m-diers]

After some tests it looks like it is fixed in version 1.9 (currently master).

[hkaiser]

Would be nice to know what was causing the problems... :/

[m-diers]

Would be nice to know what was causing the problems... :/

With the 1.9 the problems are now somehow gone. Have unfortunately no time to search the commits, which has fixed it. We also have with the 1.8.x problems with the performance whereby we are still at 1.7.1. But the performance is also with the 1.9 back to the 1.7.1 identical. Is there already a plan for the first rc of 1.9?

[hkaiser]

Would be nice to know what was causing the problems... :/

With the 1.9 the problems are now somehow gone. Have unfortunately no time to search the commits, which has fixed it. We also have with the 1.8.x problems with the performance whereby we are still at 1.7.1. But the performance is also with the 1.9 back to the 1.7.1 identical.

Good t hear we got that fixed as well.

Is there already a plan for the first rc of 1.9?

No plans yet. Perhaps early next year?