hpx icon indicating copy to clipboard operation
hpx copied to clipboard

flecsi: New hangs with HPX 1.11.0

Open rbberger opened this issue 5 months ago • 0 comments

We're observing new, consistent hangs in FleCSI in the data-index unit test when switching from HPX 1.10.0 to 1.11.0.

Our Spack environment:

spack:
  specs:
  - flecsi
  view:
    default:
      exclude:
      - gcc-runtime
      root: .spack-env/view
  packages:
    boost:
      require:
      - +debug
      - cxxstd=17 +program_options
      prefer:
      - '@1.88.0'
    c:
      prefer:
      - gcc
    cxx:
      prefer:
      - gcc
    flecsi:
      require:
      - spec: backend=hpx build_type=Debug caliper_detail=low +flog +graphviz %gcc
        when: '@2.4:'
    fortran:
      prefer:
      - gcc
    hpx:
      require:
      - build_type=Debug
      prefer:
      - '@1.11.0'
    kokkos:
      require:
      - +serial~openmp~cuda~rocm
      prefer:
      - '@4.6.01'
    mpi:
      require:
      - mpich
    cmake:
      prefer:
      - '@3.31.6'
    graphviz:
      prefer:
      - '@12.1.0'
      require:
      - +poppler+pangocairo %gcc
    py-sphinx:
      prefer:
      - '@8.2.0'
    hdf5:
      prefer:
      - '@1.14.6'
    metis:
      require:
      - '@5.1.0'
    parmetis:
      require:
      - '@4.0.3'
    mpich:
      prefer:
      - '@4.3.0'
      require:
      - +hydra +slurm +fortran device=ch4 netmod=ucx %c,cxx,fortran=gcc
    ucx:
      prefer:
      - '@1.18.0'
      require:
      - +cma+dc+rc+rdmacm+thread_multiple+ud+verbs+debug %c,cxx=gcc

Concretized:

$ spack spec
 -   [email protected]~cuda~doc+flog~format+graphviz~hdf5~ipo~rocm+shared backend=hpx build_system=cmake build_type=Debug caliper_detail=low commit=77f5313b54f6782ec2a57002ce1a9f4f35641da8 generator=make arch=linux-rhel8-broadwell %c,[email protected]
[+]      ^[email protected]+atomic~charconv+chrono~clanglibcpp~cobalt~container~context~contract~coroutine+date_time+debug+exception~fiber+filesystem+graph~graph_parallel+icu+iostreams~json+locale+log+math~mpi~mqtt5+multithreaded~nowide~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+stacktrace+system~taggedlayout+test+thread+timer~type_erasure~url~versionedlayout+wave build_system=generic cxxstd=17 patches:=a440f96 visibility=hidden arch=linux-rhel8-broadwell %c,[email protected]
[+]          ^[email protected]~debug~pic+shared build_system=generic arch=linux-rhel8-broadwell %[email protected]
[e]              ^[email protected] build_system=autotools arch=linux-rhel8-broadwell
[+]          ^[email protected] build_system=autotools cxxstd=17 arch=linux-rhel8-broadwell %c,[email protected]
[+]          ^[email protected]~pic build_system=autotools libs:=shared,static arch=linux-rhel8-broadwell %[email protected]
[+]          ^[email protected]+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rhel8-broadwell %c,[email protected]
[+]          ^[email protected]+programs build_system=makefile compression:=none libs:=shared,static arch=linux-rhel8-broadwell %c,[email protected]
[+]      ^[email protected]+adiak~cuda~fortran+gotcha~ipo+kokkos+libdw~libpfm+libunwind+mpi+papi~python~rocm+sampler+shared~sosflow~tests+tools~variorum~vtune build_system=cmake build_type=Release generator=make arch=linux-rhel8-broadwell %c,cxx,[email protected]
[+]          ^[email protected]~ipo+mpi+shared build_system=cmake build_type=Release commit=7ac997111785bee6d9391664b1d18ebc2b3c557b generator=make arch=linux-rhel8-broadwell %c,cxx,[email protected]
[+]          ^[email protected]~debuginfod+exeprefix+nls build_system=autotools arch=linux-rhel8-broadwell %c,[email protected]
[+]              ^[email protected]+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-rhel8-broadwell %c,[email protected]
[+]                  ^[email protected] build_system=autotools zip=pigz arch=linux-rhel8-broadwell %[email protected]
[+]                      ^[email protected] build_system=makefile arch=linux-rhel8-broadwell %[email protected]
[+]              ^[email protected] build_system=autotools libs:=shared,static arch=linux-rhel8-broadwell %[email protected]
[+]              ^[email protected]+sigsegv build_system=autotools arch=linux-rhel8-broadwell %c,[email protected]
[+]                  ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]          ^[email protected]~block_signals~conservative_checks~cxx_exceptions~debug~debug_frame+docs~pic+tests+weak_backtrace~xz~zlib build_system=autotools components:=none libs:=shared,static arch=linux-rhel8-broadwell %c,[email protected]
[+]          ^[email protected]~cuda~debug+example~infiniband~lmsensors~nvml~powercap~rapl+rdpmc~rocm~rocm_smi~sde+shared~static_tools build_system=autotools patches:=48cb202 arch=linux-rhel8-broadwell %c,cxx,[email protected]
[+]          ^[email protected]+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches:=13fa8bf,b0615b2,ebdca64,f2fd060 arch=linux-rhel8-broadwell %c,[email protected]
[+]              ^[email protected]+libbsd build_system=autotools arch=linux-rhel8-broadwell %c,[email protected]
[+]                  ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]                      ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]              ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]              ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %c,[email protected]
[+]              ^[email protected]~obsolete_api build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[e]              ^[email protected]~docs+shared build_system=generic certs=mozilla arch=linux-rhel8-broadwell
[+]              ^[email protected] build_system=autotools patches:=1ea4349,24f587b,3d9885e,5911a5b,622ba38,6c8adf8,758e2ec,79572ee,a177edc,bbf97f1,c7b45ff,e0013d9,e065038 arch=linux-rhel8-broadwell %[email protected]
[+]              ^[email protected]+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]              ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]      ^[email protected]~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release arch=linux-rhel8-broadwell %c,[email protected]
[e]          ^[email protected]~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs:=shared,static tls:=openssl arch=linux-rhel8-broadwell
[+]          ^[email protected]~symlinks+termlib abi=none build_system=autotools patches:=7a351bc arch=linux-rhel8-broadwell %c,[email protected]
[+]      ^[email protected] build_system=generic arch=linux-rhel8-broadwell
[e]      ^[email protected]~binutils+bootstrap~graphite~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' arch=linux-rhel8-broadwell
[+]      ^[email protected] build_system=generic arch=linux-rhel8-broadwell
[e]      ^[email protected] build_system=autotools arch=linux-rhel8-broadwell
[+]      ^[email protected]~guile build_system=generic arch=linux-rhel8-broadwell %[email protected]
[+]      ^[email protected]~doc~expat~ghostscript~gtkplus~gts~java~libgd+pangocairo+poppler~qt~quartz~x build_system=autotools arch=linux-rhel8-broadwell %c,[email protected]
[+]          ^[email protected] build_system=autotools arch=linux-rhel8-broadwell
[+]              ^[email protected]+cpanm+opcode+open+shared+threads build_system=generic arch=linux-rhel8-broadwell %[email protected]
[+]                  ^[email protected]+cxx~docs+stl build_system=autotools patches:=26090f4,b231fcc arch=linux-rhel8-broadwell %c,[email protected]
[+]          ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]          ^[email protected]~color build_system=autotools arch=linux-rhel8-broadwell %c,[email protected]
[+]          ^[email protected]~X+fc+ft+gobject+pdf+pic+png+shared~svg build_system=autotools patches:=7097196 arch=linux-rhel8-broadwell %c,[email protected]
[+]              ^[email protected]+shared~strip build_system=meson buildtype=release default_library:=shared arch=linux-rhel8-broadwell %[email protected]
[+]              ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]          ^[email protected]+lex~nls build_system=autotools arch=linux-rhel8-broadwell %c,[email protected]
[+]          ^[email protected]~pic build_system=autotools arch=linux-rhel8-broadwell %c,[email protected]
[+]              ^[email protected] build_system=autotools fonts:=encodings,font-adobe-100dpi,font-adobe-75dpi,font-adobe-utopia-100dpi,font-adobe-utopia-75dpi,font-adobe-utopia-type1,font-alias,font-arabic-misc,font-bh-100dpi,font-bh-75dpi,font-bh-lucidatypewriter-100dpi,font-bh-lucidatypewriter-75dpi,font-bh-type1,font-bitstream-100dpi,font-bitstream-75dpi,font-bitstream-speedo,font-bitstream-type1,font-cronyx-cyrillic,font-cursor-misc,font-daewoo-misc,font-dec-misc,font-ibm-type1,font-isas-misc,font-jis-misc,font-micro-misc,font-misc-cyrillic,font-misc-ethiopic,font-misc-meltho,font-misc-misc,font-mutt-misc,font-schumacher-misc,font-screen-cyrillic,font-sun-misc,font-winitzki-cyrillic,font-xfree86-type1 arch=linux-rhel8-broadwell %[email protected]
[+]                  ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]                      ^[email protected] build_system=autotools arch=linux-rhel8-broadwell
[+]                      ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]                          ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]                      ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]                  ^[email protected] build_system=autotools arch=linux-rhel8-broadwell
[+]                  ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]                      ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]              ^[email protected] build_system=autotools patches:=3dd36db arch=linux-rhel8-broadwell %c,[email protected]
[+]          ^[email protected]+pic+shared build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]          ^[email protected]~libmount~strip build_system=meson buildtype=release default_library:=shared tracing:=none arch=linux-rhel8-broadwell %c,[email protected]
[+]              ^[email protected] build_system=python_pip patches:=0f0b1bd arch=linux-rhel8-broadwell
[+]                  ^[email protected] build_system=generic arch=linux-rhel8-broadwell
[+]                  ^[email protected] build_system=generic arch=linux-rhel8-broadwell
[+]                  ^[email protected] build_system=generic arch=linux-rhel8-broadwell
[+]              ^[email protected]~jit+multibyte+pic build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]              ^[email protected] build_system=generic arch=linux-rhel8-broadwell
[+]          ^[email protected]~ipo~pic build_system=cmake build_type=Release generator=make libs:=shared,static arch=linux-rhel8-broadwell %c,[email protected]
[+]          ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]          ^[email protected]~X~strip build_system=meson buildtype=release default_library:=shared arch=linux-rhel8-broadwell %c,[email protected]
[+]              ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]              ^[email protected]~strip build_system=meson buildtype=release default_library:=shared arch=linux-rhel8-broadwell %[email protected]
[+]              ^[email protected]~graphite2~strip build_system=meson buildtype=release default_library:=shared arch=linux-rhel8-broadwell %c,[email protected]
[+]              ^[email protected]~strip build_system=meson buildtype=release default_library:=shared arch=linux-rhel8-broadwell
[+]          ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]          ^[email protected]~boost~cms~cpp~glib~gobject~gpgme~ipo~jpeg~libcurl~nss~openjpeg~png~qt~tiff~zlib build_system=cmake build_type=Release generator=make arch=linux-rhel8-broadwell %c,[email protected]
[+]              ^[email protected]~ipo build_system=cmake build_type=Release generator=make arch=linux-rhel8-broadwell
[+]          ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]      ^[email protected]~apex~async_cuda~async_mpi~cuda~examples~generic_coroutines~ipo~rocm~tools build_system=cmake build_type=Debug cxxstd=17 generator=ninja instrumentation:=none malloc=system max_cpu_count=auto networking:=mpi arch=linux-rhel8-broadwell %[email protected]
[+]          ^[email protected]~boost_coroutine~boost_regex~separate_compilation build_system=autotools cxxstd=17 arch=linux-rhel8-broadwell %c,[email protected]
[e]          ^[email protected]+man+nls+perl+subtree~svn~tcltk build_system=autotools arch=linux-rhel8-broadwell
[+]          ^[email protected]~cairo~cuda~gl~level_zero~libudev+libxml2~nvml~opencl+pci~rocm build_system=autotools libs:=shared,static arch=linux-rhel8-broadwell %c,[email protected]
[+]          ^[email protected]+re2c build_system=generic patches:=93f4bb3 arch=linux-rhel8-broadwell %c,[email protected]
[+]              ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %c,[email protected]
[+]      ^[email protected]~aggressive_vectorization~cmake_lang~compiler_warnings+complex_align~cuda~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hip_relocatable_device_code~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic~rocm+serial+shared~sycl~tests~threads~tuning~wrapper build_system=cmake build_type=Release cxxstd=17 generator=make intel_gpu_arch=none arch=linux-rhel8-broadwell %c,[email protected]
[+]      ^[email protected]~argobots~cuda+fortran~hcoll+hwloc+hydra~level_zero+libxml2+pci~rocm+romio+slurm~vci~verbs+wrapperrpath~xpmem build_system=autotools datatype-engine=auto device=ch4 netmod=ucx pmi=default arch=linux-rhel8-broadwell %c,cxx,[email protected]
[+]          ^[email protected] build_system=autotools patches:=440b954 arch=linux-rhel8-broadwell %[email protected]
[+]          ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]              ^[email protected] build_system=autotools arch=linux-rhel8-broadwell
[+]          ^[email protected]~http+pic~python+shared build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[e]          ^slurm@24-05-5-1~cgroup~gtk~hdf5~hwloc~mariadb~nvml~pam~pmix+readline~restd~rsmi build_system=autotools sysconfdir=PREFIX/etc arch=linux-rhel8-broadwell
[+]          ^[email protected]~assertions~backtrace_detail+cma~cuda+dc+debug~dm+examples~gdrcopy~gtest~ib_hw_tm~java~knem~logging~mlx5_dv+openmp+optimizations~parameter_checking+pic+rc+rdmacm~rocm+thread_multiple~ucg+ud+verbs~vfs~xpmem build_system=autotools libs:=shared,static opt=3 simd:=auto arch=linux-rhel8-broadwell %c,[email protected]
[+]              ^[email protected]~ipo+man_pages+pyverbs+static build_system=cmake build_type=Release generator=make arch=linux-rhel8-broadwell %[email protected]
[+]                  ^[email protected] build_system=autotools arch=linux-rhel8-broadwell %[email protected]
[+]                  ^[email protected] build_system=python_pip arch=linux-rhel8-broadwell
[+]          ^[email protected]~cuda~level_zero~rocm build_system=autotools arch=linux-rhel8-broadwell %[email protected]

rbberger avatar Jul 16 '25 00:07 rbberger