Static build on Summit issues

[pnorbert]

Two issues with -DBUILD_SHARED_LIBS=OFF on Summit

• dependencies are still dynamic libraries but we need the static dependencies
• libfabric is not configured even though the library is found and its static version exists

summit:/ccs/proj/e2e/pnorbert/ADIOS/ADIOS2/build.summit.master.pgi
$ module list

Currently Loaded Modules:
  1) lsf-tools/2.0                4) xalt/1.2.1   7) git/2.31.1                      10) python/3.8-anaconda3      13) zfp/0.5.5
  2) hsi/5.0.2.p5                 5) DefApps      8) spectrum-mpi/10.4.0.3-20210112  11) libfabric/1.12.1-sysrdma
  3) darshan-runtime/3.3.0-lite   6) pgi/20.4     9) cmake/3.21.3                    12) c-blosc/1.21.0

summit:/ccs/proj/e2e/pnorbert/ADIOS/ADIOS2/build.summit.master.pgi
$ cmake -DCMAKE_INSTALL_PREFIX=/ccs/proj/e2e/pnorbert/ADIOS/ADIOS2/build.summit.master.pgi/install -DCMAKE_CXX_COMPILER=pgc++ -DCMAKE_C_COMPILER=pgcc -DCMAKE_Fortran_COMPILER=pgfortran -DBUILD_SHARED_LIBS=OFF -DADIOS2_USE_MPI=ON -DADIOS2_USE_Fortran=ON -DADIOS2_BUILD_EXAMPLES_EXPERIMENTAL=OFF -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF -DADIOS2_USE_DataMan=OFF -DADIOS2_USE_Python=OFF -DCMAKE_PREFIX_PATH=/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-8.3.1/c-blosc-1.21.0-kn4fxiiluv4pfjdihewe5wbnkt4urjxa -DADIOS2_USE_HDF5=OFF ..
-- The C compiler identification is PGI 20.4.0
-- The CXX compiler identification is PGI 20.4.0
...
-- Found Blosc: /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-8.3.1/c-blosc-1.21.0-kn4fxiiluv4pfjdihewe5wbnkt4urjxa/lib/libblosc.so (found suitable version "1.21.0", minimum required is "1.7") 
-- Found BZip2: /usr/lib64/libbz2.so (found version "1.0.6") 
...
-- Checking for module 'libfabric'
--   Found libfabric, version 1.12.1
-- Could NOT find LIBFABRIC (missing: LIBFABRIC_LIBRARIES) (Required is at least version "1.6")
...
ADIOS2 build configuration:
  ADIOS Version: 2.7.1.1304
  C++ Compiler : PGI 20.4.0 
    /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-8-os/pgi-20.4-oy4x2h7acugasgx3qra4ue2mo37feobv/linuxpower/20.4/bin/pgc++

  Fortran Compiler : PGI 20.4.0 
    /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-8-os/pgi-20.4-oy4x2h7acugasgx3qra4ue2mo37feobv/linuxpower/20.4/bin/pgfortran

  Installation prefix: /ccs/proj/e2e/pnorbert/ADIOS/ADIOS2/build.summit.master.pgi/install
        bin: bin
        lib: lib64
    include: include
      cmake: lib64/cmake/adios2

  Features:
    Library Type: static (with PIC)
    Build Type:   Release
    Testing: OFF
    Examples: ON
    Build Options:
      BP5       : ON
      DataMan   : OFF
      DataSpaces: OFF
      HDF5      : OFF
      HDF5_VOL  : OFF
      MHS       : ON
      SST       : ON
      CUDA      : OFF
      Fortran   : ON
      MPI       : ON
      Python    : OFF
      Blosc     : ON
      BZip2     : ON
      LIBPRESSIO: OFF
      MGARD     : OFF
      PNG       : ON
      SZ        : OFF
      ZFP       : ON
      DAOS      : OFF
      IME       : OFF
      O_DIRECT  : ON
      SysVShMem : ON
      ZeroMQ    : OFF
      Profiling : OFF
      Endian_Reverse: OFF
      Sodium    : OFF
    RDMA Transport for Staging: Unconfigured

Libfabric is found but the .so library is picked, and then later something goes wrong and libfabric is turned off.

$ grep -i fabric CMakeCache.txt 
pkgcfg_lib_PC_LIBFABRIC_fabric:FILEPATH=/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-8.3.1/libfabric-1.12.1-fdmjhbhc3xfp4ufu3yhibnr5vy3ohsyh/lib/libfabric.so
PC_LIBFABRIC_CFLAGS:INTERNAL=
PC_LIBFABRIC_CFLAGS_I:INTERNAL=
PC_LIBFABRIC_CFLAGS_OTHER:INTERNAL=
PC_LIBFABRIC_FOUND:INTERNAL=1
PC_LIBFABRIC_INCLUDEDIR:INTERNAL=/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-8.3.1/libfabric-1.12.1-fdmjhbhc3xfp4ufu3yhibnr5vy3ohsyh/include
PC_LIBFABRIC_INCLUDE_DIRS:INTERNAL=
PC_LIBFABRIC_LDFLAGS:INTERNAL=-lfabric
PC_LIBFABRIC_LDFLAGS_OTHER:INTERNAL=
PC_LIBFABRIC_LIBDIR:INTERNAL=/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-8.3.1/libfabric-1.12.1-fdmjhbhc3xfp4ufu3yhibnr5vy3ohsyh/lib
PC_LIBFABRIC_LIBRARIES:INTERNAL=fabric
PC_LIBFABRIC_LIBRARY_DIRS:INTERNAL=
PC_LIBFABRIC_LIBS:INTERNAL=
PC_LIBFABRIC_LIBS_L:INTERNAL=
PC_LIBFABRIC_LIBS_OTHER:INTERNAL=
PC_LIBFABRIC_LIBS_PATHS:INTERNAL=
PC_LIBFABRIC_MODULE_NAME:INTERNAL=libfabric
PC_LIBFABRIC_PREFIX:INTERNAL=/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-8.3.1/libfabric-1.12.1-fdmjhbhc3xfp4ufu3yhibnr5vy3ohsyh
PC_LIBFABRIC_STATIC_CFLAGS:INTERNAL=
PC_LIBFABRIC_STATIC_CFLAGS_I:INTERNAL=
PC_LIBFABRIC_STATIC_CFLAGS_OTHER:INTERNAL=
PC_LIBFABRIC_STATIC_INCLUDE_DIRS:INTERNAL=
PC_LIBFABRIC_STATIC_LDFLAGS:INTERNAL=-lfabric;-lrt;-lrdmacm;-libverbs;-latomic;-lpthread;-ldl
PC_LIBFABRIC_STATIC_LDFLAGS_OTHER:INTERNAL=
PC_LIBFABRIC_STATIC_LIBDIR:INTERNAL=
PC_LIBFABRIC_STATIC_LIBRARIES:INTERNAL=fabric;rt;rdmacm;ibverbs;atomic;pthread;dl
PC_LIBFABRIC_STATIC_LIBRARY_DIRS:INTERNAL=
PC_LIBFABRIC_STATIC_LIBS:INTERNAL=
PC_LIBFABRIC_STATIC_LIBS_L:INTERNAL=
PC_LIBFABRIC_STATIC_LIBS_OTHER:INTERNAL=
PC_LIBFABRIC_STATIC_LIBS_PATHS:INTERNAL=
PC_LIBFABRIC_VERSION:INTERNAL=1.12.1
PC_LIBFABRIC_libfabric_INCLUDEDIR:INTERNAL=
PC_LIBFABRIC_libfabric_LIBDIR:INTERNAL=
PC_LIBFABRIC_libfabric_PREFIX:INTERNAL=
PC_LIBFABRIC_libfabric_VERSION:INTERNAL=
__pkg_config_arguments_PC_LIBFABRIC:INTERNAL=libfabric
__pkg_config_checked_PC_LIBFABRIC:INTERNAL=1
//ADVANCED property for variable: pkgcfg_lib_PC_LIBFABRIC_fabric
pkgcfg_lib_PC_LIBFABRIC_fabric-ADVANCED:INTERNAL=1


$ ls -l /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-8.3.1/libfabric-1.12.1-fdmjhbhc3xfp4ufu3yhibnr5vy3ohsyh/lib
total 3048
-rw-r--r-- 1 belhorn ccsstaff 1939700 Aug 24  2021 libfabric.a
lrwxrwxrwx 1 belhorn ccsstaff      19 Aug 24  2021 libfabric.so -> libfabric.so.1.15.1
lrwxrwxrwx 1 belhorn ccsstaff      19 Aug 24  2021 libfabric.so.1 -> libfabric.so.1.15.1
-rwxr-xr-x 1 belhorn ccsstaff 1158448 Aug 24  2021 libfabric.so.1.15.1
drwxr-sr-x 2 belhorn ccsstaff    4096 Aug 24  2021 pkgconfig

blosc is found but the .so library is picked and used

$ grep blosc CMakeCache.txt 
BLOSC_INCLUDE_DIR:PATH=/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-8.3.1/c-blosc-1.21.0-kn4fxiiluv4pfjdihewe5wbnkt4urjxa/include
BLOSC_LIBRARY:FILEPATH=/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-8.3.1/c-blosc-1.21.0-kn4fxiiluv4pfjdihewe5wbnkt4urjxa/lib/libblosc.so
CMAKE_PREFIX_PATH:UNINITIALIZED=/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-8.3.1/c-blosc-1.21.0-kn4fxiiluv4pfjdihewe5wbnkt4urjxa
FIND_PACKAGE_MESSAGE_DETAILS_Blosc:INTERNAL=[/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-8.3.1/c-blosc-1.21.0-kn4fxiiluv4pfjdihewe5wbnkt4urjxa/lib/libblosc.so][/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-8.3.1/c-blosc-1.21.0-kn4fxiiluv4pfjdihewe5wbnkt4urjxa/include][v1.21.0(1.7)]

$ ll /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-8.3.1/c-blosc-1.21.0-kn4fxiiluv4pfjdihewe5wbnkt4urjxa/lib
total 516
drwxr-sr-x 2 belhorn ccsstaff   4096 Oct  9 00:16 pkgconfig/
-rw-r--r-- 1 belhorn ccsstaff 296930 Oct  9 00:16 libblosc.a
lrwxrwxrwx 1 belhorn ccsstaff     13 Oct  9 00:16 libblosc.so -> libblosc.so.1*
lrwxrwxrwx 1 belhorn ccsstaff     18 Oct  9 00:16 libblosc.so.1 -> libblosc.so.1.21.0*
-rwxr-xr-x 1 belhorn ccsstaff 215504 Oct  9 00:16 libblosc.so.1.21.0*

[pnorbert]

ping @jychoi-hpc since we need this for XGC at 4k runs. There is trouble loading the shared libraries into 24600 processes.

[eisenhauer]

@pnorbert Interesting. I wonder if the DLL loading might be part of why I'm having trouble getting test runs at scale for SST... Let me look at the libfabric aspects.

[eisenhauer]

OK, I'm going to have to punt on this. I think the answer is that both the FindLIBFABRIC.cmake and FindBlosc.cmake are insufficient to support static linking of those libraries. The same can likely be said for FindCrayDRC.cmake and DataSpaces, as they are structured like the FindLIBFABRIC.cmake. I can't speak for the other Find*.cmake files in ADIOS2. @chuckatkins, just exactly how this is supposed to work, and whether or not we're at the mercy of appropriate 'tagging' in the pkg-conf file created by the libraries in question isn't clear to me, so kicking it in your direction. There are a lot of Find*.cmake things out there that purport to support static linking, but they seem to be a lot more complex than these files. Hoping you've got better info than I do.

[pnorbert]

Use -static in the flags during cmake to make this work CFLAGS="-static" CXXFLAGS="-static" FCFLAGS="-static" This will forcibly turn off -DBUILD_SHARED_LIBS and will not allow to find any dynamic library.

[vicentebolea]

Possibly solved with #3715