Investigate CUDA compilation problems

[hkaiser]

Two of our CUDA tests fail compiling by NVCC unceremoniously stopping without error message. For now, these tests have been disabled (see: https://github.com/STEllAR-GROUP/hpx/pull/5796). We should investigate whether this can be worked around.

[hkaiser]

@G-071 Gregor, this is the ticket I promised to create.

[G-071]

NVCC apparently got some improvement/fixes with the CUDA/12.3 release meaning we might be able to finally fix this old issue (which is still very much relevant as the same problem prevents us from using/compiling Kokkos >= 4.1 with gcc and the HPX and CUDA execution space both enabled)

With previous CUDA releases, it simply crashed when compiling transform_stream.cu: nvcc error : 'cudafe++' died due to signal 11 (Invalid memory reference)

With CUDA 12.3 we get some actual compilation errors instead:

[100%] Building CUDA object libs/core/async_cuda/tests/unit/CMakeFiles/transform_stream_test.dir/transform_stream.cu.o
cd /work/gdaiss/hpx/build/libs/core/async_cuda/tests/unit && /usr/local/cuda-12.3/bin/nvcc -forward-unknown-to-host-compiler -DASIO_DISABLE_CONSTEXPR -DHPX_APPLICATION_EXPORTS -DHPX_APPLICATION_NAME=transform_stream_test -DHPX_APPLICATION_NAME_DEFAULT=transform_stream_test -DHPX_APPLICATION_STRING=\"transform_stream_test\" -DHPX_DEBUG -DHPX_PREFIX=\"/work/gdaiss/hpx/build\" -DHPX_PREFIX_DEFAULT=\"/usr/local\" -D_GNU_SOURCE --options-file CMakeFiles/transform_stream_test.dir/includes_CUDA.rsp -g -std=c++17 --generate-code=arch=compute_70,code=[compute_70,sm_70] --generate-code=arch=compute_80,code=[compute_80,sm_80] -Xcompiler=-fPIE
 -w --extended-lambda --default-stream per-thread --expt-relaxed-constexpr -MD -MT libs/core/async_cuda/tests/unit/CMakeFiles/transform_stream_test.dir/transform_stream.cu.o -MF CMakeFiles/transform_stream_test.dir/transform_stream.cu.o.d -x cu -c /work/gdaiss/hpx/libs/core/async_cuda/tests/unit/transform_stream.cu -o CMakeFiles/transform_stream_test.dir/transform_stream.cu.o                                                                                                                                                                                                                                                                    
/work/gdaiss/hpx/libs/core/execution/include/hpx/execution/algorithms/just.hpp(106): error: expected a ">"
                      all_of_v<std::is_nothrow_move_constructible<Ts>...>)
                                                                     ^
          detected during:
            instantiation of "hpx::execution::experimental::detail::tag_invoke" based on template argument <hpx::cuda::experimental::detail::transform_stream_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::tra
nsform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver, dummy>> at line 79 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/traits/is_invocable.hpp                                                                                                                                                                                                                                
            instantiation of class "hpx::is_invocable<F, Ts...> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t, Ts=<hpx::execution::experimental::connect_t, hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>> &&, hpx::cuda::experimental::detail::transform_stream_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::
thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver, dummy> &&>]" at line 167 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/tag_invoke.hpp                                   
            instantiation of "const __nv_bool hpx::functional::is_tag_invocable_v [with Tag=hpx::execution::experimental::connect_t, Args=<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>> &&, hpx::cuda::experimental::detail::transform_stream_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>::operatio
n_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver, dummy> &&>]" at line 309 of /work/gdaiss/hpx/libs/core/async_cuda/include/hpx/async_cuda/transform_stream.hpp                                                             
            instantiation of "auto hpx::cuda::experimental::detail::tag_invoke(hpx::execution::experimental::connect_t, hpx::cuda::experimental::detail::transform_stream_sender<std::remove_reference<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::make_index_pack_t<0UL>> &>::type, dummy> &&, R &&) [with R=hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimenta
l::thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver]" at line 79 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/traits/is_invocable.hpp                                    
            instantiation of class "hpx::is_invocable<F, Ts...> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t, Ts=<hpx::execution::experimental::connect_t, hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy> &&, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experime
ntal::thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver &&>]" at line 167 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/tag_invoke.hpp                                     
            [ 6 instantiation contexts not shown ]
            instantiation of class "hpx::util::detail::invoke_result_impl<F (Ts...), std::void_t<decltype((<expression>))>> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t &&, Ts=<hpx::execution::experimental::connect_t &&, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&, hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::
schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type &&>]" at line 38 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/invoke_result.hpp                                                                                                                                                                                                              
            instantiation of class "hpx::util::invoke_result<F, Ts...> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t, Ts=<hpx::execution::experimental::connect_t, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&, hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::trans
form_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type &&>]" at line 443 of /work/gdaiss/hpx/libs/core/execution/include/hpx/execution/algorithms/sync_wait.hpp                                                                                                                                                                                                                                                                   
            instantiation of "auto hpx::this_thread::experimental::tag_fallback_invoke(hpx::this_thread::experimental::sync_wait_t, Sender &&) [with Sender=hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, _concept_requires_427=42, _concept_check_427=0]" at line 79 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/traits/is_invocable.hpp                               
            instantiation of class "hpx::is_invocable<F, Ts...> [with F=const hpx::functional::detail::tag_fallback_invoke_t_ns::tag_fallback_invoke_t, Ts=<hpx::this_thread::experimental::sync_wait_t, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&>]" at line 171 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/detail/tag_fallback_invoke.hpp                      
            instantiation of "const __nv_bool hpx::functional::detail::is_tag_fallback_invocable_v [with Tag=hpx::this_thread::experimental::sync_wait_t, Args=<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&>]" at line 125 of /work/gdaiss/hpx/libs/core/async_cuda/tests/unit/transform_stream.cu                                                                                      

/work/gdaiss/hpx/libs/core/execution/include/hpx/execution/algorithms/just.hpp(106): error: pack expansion does not make use of any argument packs
                      all_of_v<std::is_nothrow_move_constructible<Ts>...>)
                                                                     ^
          detected during:
            instantiation of class "hpx::is_invocable<F, Ts...> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t, Ts=<hpx::execution::experimental::connect_t, hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>> &&, hpx::cuda::experimental::detail::transform_stream_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::
thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver, dummy> &&>]" at line 167 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/tag_invoke.hpp                                   
            instantiation of "const __nv_bool hpx::functional::is_tag_invocable_v [with Tag=hpx::execution::experimental::connect_t, Args=<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>> &&, hpx::cuda::experimental::detail::transform_stream_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>::operatio
n_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver, dummy> &&>]" at line 309 of /work/gdaiss/hpx/libs/core/async_cuda/include/hpx/async_cuda/transform_stream.hpp                                                             
            instantiation of "auto hpx::cuda::experimental::detail::tag_invoke(hpx::execution::experimental::connect_t, hpx::cuda::experimental::detail::transform_stream_sender<std::remove_reference<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::make_index_pack_t<0UL>> &>::type, dummy> &&, R &&) [with R=hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimenta
l::thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver]" at line 79 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/traits/is_invocable.hpp                                    
            instantiation of class "hpx::is_invocable<F, Ts...> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t, Ts=<hpx::execution::experimental::connect_t, hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy> &&, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experime
ntal::thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver &&>]" at line 167 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/tag_invoke.hpp                                     
            instantiation of "const __nv_bool hpx::functional::is_tag_invocable_v [with Tag=hpx::execution::experimental::connect_t, Args=<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy> &&, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>::op
eration_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver &&>]" at line 38 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/invoke_result.hpp                                                                   
            [ 5 instantiation contexts not shown ]
            instantiation of class "hpx::util::detail::invoke_result_impl<F (Ts...), std::void_t<decltype((<expression>))>> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t &&, Ts=<hpx::execution::experimental::connect_t &&, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&, hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::
schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type &&>]" at line 38 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/invoke_result.hpp                                                                                                                                                                                                              
            instantiation of class "hpx::util::invoke_result<F, Ts...> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t, Ts=<hpx::execution::experimental::connect_t, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&, hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::trans
form_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type &&>]" at line 443 of /work/gdaiss/hpx/libs/core/execution/include/hpx/execution/algorithms/sync_wait.hpp                                                                                                                                                                                                                                                                   
            instantiation of "auto hpx::this_thread::experimental::tag_fallback_invoke(hpx::this_thread::experimental::sync_wait_t, Sender &&) [with Sender=hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, _concept_requires_427=42, _concept_check_427=0]" at line 79 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/traits/is_invocable.hpp                               
            instantiation of class "hpx::is_invocable<F, Ts...> [with F=const hpx::functional::detail::tag_fallback_invoke_t_ns::tag_fallback_invoke_t, Ts=<hpx::this_thread::experimental::sync_wait_t, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&>]" at line 171 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/detail/tag_fallback_invoke.hpp                      
            instantiation of "const __nv_bool hpx::functional::detail::is_tag_fallback_invocable_v [with Tag=hpx::this_thread::experimental::sync_wait_t, Args=<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&>]" at line 125 of /work/gdaiss/hpx/libs/core/async_cuda/tests/unit/transform_stream.cu                                                                                      

/work/gdaiss/hpx/libs/core/type_support/include/hpx/type_support/pack.hpp(125): error: class "hpx::util::all_of<std::is_nothrow_move_constructible<<error-type>>>" has no member "value"
      inline constexpr bool all_of_v = all_of<Ts...>::value;
                                                      ^
          detected during:
            instantiation of "const __nv_bool hpx::util::all_of_v [with Ts=<std::is_nothrow_move_constructible<<error-type>>>]" at line 106 of /work/gdaiss/hpx/libs/core/execution/include/hpx/execution/algorithms/just.hpp
            instantiation of class "hpx::is_invocable<F, Ts...> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t, Ts=<hpx::execution::experimental::connect_t, hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>> &&, hpx::cuda::experimental::detail::transform_stream_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::
thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver, dummy> &&>]" at line 167 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/tag_invoke.hpp                                   
            instantiation of "const __nv_bool hpx::functional::is_tag_invocable_v [with Tag=hpx::execution::experimental::connect_t, Args=<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>> &&, hpx::cuda::experimental::detail::transform_stream_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>::operatio
n_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver, dummy> &&>]" at line 309 of /work/gdaiss/hpx/libs/core/async_cuda/include/hpx/async_cuda/transform_stream.hpp                                                             
            instantiation of "auto hpx::cuda::experimental::detail::tag_invoke(hpx::execution::experimental::connect_t, hpx::cuda::experimental::detail::transform_stream_sender<std::remove_reference<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::make_index_pack_t<0UL>> &>::type, dummy> &&, R &&) [with R=hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimenta
l::thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver]" at line 79 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/traits/is_invocable.hpp                                    
            instantiation of class "hpx::is_invocable<F, Ts...> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t, Ts=<hpx::execution::experimental::connect_t, hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy> &&, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experime
ntal::thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver &&>]" at line 167 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/tag_invoke.hpp                                     
            [ 6 instantiation contexts not shown ]
            instantiation of class "hpx::util::detail::invoke_result_impl<F (Ts...), std::void_t<decltype((<expression>))>> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t &&, Ts=<hpx::execution::experimental::connect_t &&, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&, hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::
schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type &&>]" at line 38 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/invoke_result.hpp                                                                                                                                                                                                              
            instantiation of class "hpx::util::invoke_result<F, Ts...> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t, Ts=<hpx::execution::experimental::connect_t, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&, hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::trans
form_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type &&>]" at line 443 of /work/gdaiss/hpx/libs/core/execution/include/hpx/execution/algorithms/sync_wait.hpp                                                                                                                                                                                                                                                                   
            instantiation of "auto hpx::this_thread::experimental::tag_fallback_invoke(hpx::this_thread::experimental::sync_wait_t, Sender &&) [with Sender=hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, _concept_requires_427=42, _concept_check_427=0]" at line 79 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/traits/is_invocable.hpp                               
            instantiation of class "hpx::is_invocable<F, Ts...> [with F=const hpx::functional::detail::tag_fallback_invoke_t_ns::tag_fallback_invoke_t, Ts=<hpx::this_thread::experimental::sync_wait_t, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&>]" at line 171 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/detail/tag_fallback_invoke.hpp                      
            instantiation of "const __nv_bool hpx::functional::detail::is_tag_fallback_invocable_v [with Tag=hpx::this_thread::experimental::sync_wait_t, Args=<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&>]" at line 125 of /work/gdaiss/hpx/libs/core/async_cuda/tests/unit/transform_stream.cu                                                                                      

3 errors detected in the compilation of "/work/gdaiss/hpx/libs/core/async_cuda/tests/unit/transform_stream.cu".
make[3]: *** [libs/core/async_cuda/tests/unit/CMakeFiles/transform_stream_test.dir/build.make:77: libs/core/async_cuda/tests/unit/CMakeFiles/transform_stream_test.dir/transform_stream.cu.o] Error 1
                                                                                                                                                                                                                                                                    ```

To reproduce:

1. Log into one of the cuda-A100-intel nodes on rostam
2. module load cuda/12.3 gcc/12.3
3. cmake step:

cmake -DCMAKE_BUILD_TYPE=Debug -DHPX_WITH_PARALLEL_TESTS_BIND_NONE=ON  -DHPX_WITH_CHECK_MODULE_DEPENDENCIES=ON -DHPX_WITH_PARCELPORT_COUNTERS=ON -DHPX_WITH_CXX_STANDARD=17 -DHPX_WITH_MALLOC=system -DHPX_WITH_FETCH_ASIO=ON -DHPX_WITH_COMPILER_WARNI
NGS=ON -DHPX_WITH_COMPILER_WARNINGS_AS_ERRORS=OFF -DHPX_WITH_CUDA=ON -DHPX_WITH_NETWORKING=OFF -DHPX_WITH_DISTRIBUTED_RUNTIME=OFF  -DHPX_WITH_ASYNC_MPI=OFF -DCMAKE_CUDA_ARCHITECTURES='70;80'

(Note, changing the CXX standard to 20 does not seem change anything about this issue -- we still get the same error message) 4. make tests.unit.modules.async_cuda.transform_stream VERBOSE=1 -j32

I have tested this both with the current HPX master and with the commit I tried last year (when last checking this with CUDA/12.1), just to make sure this new behavior is not due to some changes in HPX (it is not -- without CUDA/12.3 I can still reproduce the old nvcc crash).

@hkaiser Given the new compilation errors: do you have an ideas how to fix this (or work around it)?

[hkaiser]

@G-071 we could do the same as we've done in similar cases before, i.e. wrap the whole noexcept() clause into a #if !defined(HPX_COMPUTE_DEVICE_CODE)