Build issue with CMake (linking to lapack?) using master

[osborn9]

This issue came up using spack to build superlu-dist@develop (as part of hypre's regression testing). Here's the spack-build-out.txt for spack install superlu-dist@develop using linux-rhel7-sandybridge architecture with [email protected] and [email protected].

Commit cba813d044fc3f848a0a59b7992b07ff64ff9a6b seemingly resolved the issue, yet it re-appeared after 1a6d5bad87736c9e6f53530a63dce79856ecd76c.

Please let me know if I can provide more details. Thanks! Sarah

@ulrikeyang

[xiaoyeli]

We just had a big release. Can you please let me know what are the errors? It's not clear from those 2 commits.

Thanks, Sherry

[osborn9]

Sure, the full verbose output from the spack build is in this file: spack-build-out.txt

Then here's a snippet: [ 94%] Linking C executable pddrive4 cd /tmp/osborn9/spack-stage/spack-stage-superlu-dist-develop-2rhvo6wewssxedna4hfioae5vg77pwp3/spack-build-2rhvo6w/EXAMPLE && /home/osborn9/Code/spack/opt/spack/linux-rhel7-sandybridge/gcc-4.8.5/cmake-3.20.2-omwapc6afx7ublxrcpj7fwydzt4nhg67/bin/cmake -E cmake_link_script CMakeFiles/pddrive4.dir/link.txt --verbose=1 ../SRC/libsuperlu_dist.so.7.0.0: undefined reference to 'dgemm' ../SRC/libsuperlu_dist.so.7.0.0: undefined reference to 'dscal' ../SRC/libsuperlu_dist.so.7.0.0: undefined reference to 'daxpy' ../SRC/libsuperlu_dist.so.7.0.0: undefined reference to 'dtrsv' ../SRC/libsuperlu_dist.so.7.0.0: undefined reference to 'dtrsm' ../SRC/libsuperlu_dist.so.7.0.0: undefined reference to 'dgemv' ../SRC/libsuperlu_dist.so.7.0.0: undefined reference to 'dger' collect2: error: ld returned 1 exit status make[2]: *** [EXAMPLE/pddrive_spawn] Error 1

[ulrikeyang]

Hi Sherry, Satish has the same problems when trying to build xSDK subsets with SuperLU_dist. Ulrike

From: Sarah Osborn @.> Sent: Monday, May 17, 2021 3:33 PM To: xiaoyeli/superlu_dist @.> Cc: Yang, Ulrike Meier @.>; Mention @.> Subject: Re: [xiaoyeli/superlu_dist] Build issue with CMake (linking to lapack?) using master (#73)

Sure, the full verbose output from the spack build is in this file: spack-build-out.txthttps://urldefense.us/v3/__https:/github.com/xiaoyeli/superlu_dist/files/6497179/spack-build-out.txt__;!!G2kpM7uM-TzIFchu!gFHQipOn4AE9I1oARckz0jlASb5qiBy-c3Ju-aVFjO5_pMF04Q42aZ9SmEdgtB36$

Then here's a snippet: [ 94%] Linking C executable pddrive4 cd /tmp/osborn9/spack-stage/spack-stage-superlu-dist-develop-2rhvo6wewssxedna4hfioae5vg77pwp3/spack-build-2rhvo6w/EXAMPLE && /home/osborn9/Code/spack/opt/spack/linux-rhel7-sandybridge/gcc-4.8.5/cmake-3.20.2-omwapc6afx7ublxrcpj7fwydzt4nhg67/bin/cmake -E cmake_link_script CMakeFiles/pddrive4.dir/link.txt --verbose=1 ../SRC/libsuperlu_dist.so.7.0.0: undefined reference to 'dgemm' ../SRC/libsuperlu_dist.so.7.0.0: undefined reference to 'dscal' ../SRC/libsuperlu_dist.so.7.0.0: undefined reference to 'daxpy' ../SRC/libsuperlu_dist.so.7.0.0: undefined reference to 'dtrsv' ../SRC/libsuperlu_dist.so.7.0.0: undefined reference to 'dtrsm' ../SRC/libsuperlu_dist.so.7.0.0: undefined reference to 'dgemv' ../SRC/libsuperlu_dist.so.7.0.0: undefined reference to 'dger' collect2: error: ld returned 1 exit status make[2]: *** [EXAMPLE/pddrive_spawn] Error 1

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.us/v3/__https:/github.com/xiaoyeli/superlu_dist/issues/73*issuecomment-842685870__;Iw!!G2kpM7uM-TzIFchu!gFHQipOn4AE9I1oARckz0jlASb5qiBy-c3Ju-aVFjO5_pMF04Q42aZ9SmAqpuqCd$, or unsubscribehttps://urldefense.us/v3/__https:/github.com/notifications/unsubscribe-auth/AD4NLLLKOIN7JLELZ6FP4WLTOGKTLANCNFSM45A3JNNQ__;!!G2kpM7uM-TzIFchu!gFHQipOn4AE9I1oARckz0jlASb5qiBy-c3Ju-aVFjO5_pMF04Q42aZ9SmMT83-TZ$.

[xiaoyeli]

@balay Satish: the error is related BLAS Fortran routines requiring underscorre_, it should be handled in this header file SRC/superlu_FortranCInterface.h, these 2 lines:

/* Mangling for Fortran global symbols without underscores. */ #define FC_GLOBAL(name,NAME) name

/* Mangling for Fortran global symbols with underscores. */ #define FC_GLOBAL_(name,NAME) name

I believe the correct one should be:

#define FC_GLOBAL(name,NAME) name##_ #define FC_GLOBAL_(name,NAME) name##_

Do you agree? (I borrowed this from some other codes. I don't understand how my last commit stripped off the ending "##_"

cc: @osborn9 @ulrikeyang

[balay]

This works - but this is strange.

There appears to be some basic issue with cmake name mangling detection - and that is used in superlu-dist sources.

https://cmake.org/cmake/help/latest/module/FortranCInterface.html suggests SRC/superlu_FortranCInterface.h gets generated by cmake [based on fortran compiler - it detects]

but in this case - it appears to be in the git repo - i.e its not regenerated by cmake. So I don't really understand what the primary issue here is.

[balay]

And this is strange

if (XSDK_ENABLE_Fortran)
  include(FortranCInterface)
  FortranCInterface_HEADER(${SuperLU_DIST_SOURCE_DIR}/SRC/superlu_FortranCInterface.h
  MACRO_NAMESPACE "FC_")
  FortranCInterface_VERIFY(CXX)
  SET(MPI_Fortran_LINK_FLAGS "${CMAKE_EXE_LINKER_FLAGS}")
endif()

SRC/superlu_FortranCInterface.h should always get used [for blas/lapack] - so shouldn't this code detection get always done [when fortran is enabled] - i.e having this code under XSDK_ENABLE_Fortran flag doesn't look correct.

[xiaoyeli]

Thanks for pointing this out. Now I understand the logic. I believe that I was testing the FORTRAN/ wrapper on Summit and my laptop. Summit generates: #define FC_GLOBAL(name,NAME) name but my laptop generates: #define FC_GLOBAL(name,NAME) name##_

So the last commit was due to testting output from Summit, which generated "without-underscore".

But since you are not enabling Fortran, then, you are using my last commit, and it is inconsistent with your compiler system.

I will remove the if (XSDK_ENABLE_Fortran) test, and always generates superlu_FortranCInterface.h Do you agree with this?

However, in the early version 6.4.0: https://github.com/xiaoyeli/superlu_dist/blob/v6.4.0/CMakeLists.txt This if-test also existed. I guess it was just by chance that the output I committed was from my laptop, having underscore, and it is consistent with most machines.

@osborn9 @ulrikeyang

[balay]

I see XSDK_ENABLE_Fortran is used in multiple places - so just removing that single if might not work. (there is some inconsistency below - some text says ON and some says TRUE)

$ git grep XSDK_ENABLE_Fortran
CMakeLists.txt:if (XSDK_ENABLE_Fortran)
CMakeLists.txt:if (XSDK_ENABLE_Fortran)
CMakeLists.txt:if (XSDK_ENABLE_Fortran)
CMakeLists.txt:if (XSDK_ENABLE_Fortran)
CMakeLists.txt:if (XSDK_ENABLE_Fortran)
README.md:    -DXSDK_ENABLE_Fortran=OFF | ON
README.md:  -DXSDK_ENABLE_Fortran=OFF \
cmake/XSDKDefaults.cmake:# XSDK_ENABLE_C=OFF, XSDK_ENABLE_CXX=OFF, or XSDK_ENABLE_Fortran=OFF *before*
cmake/XSDKDefaults.cmake:#   SET(XSDK_ENABLE_Fortran OFF)
cmake/XSDKDefaults.cmake:SET(XSDK_ENABLE_Fortran  TRUE)
cmake/XSDKDefaults.cmake:  IF (XSDK_ENABLE_Fortran)
make.inc.in:XSDK_ENABLE_Fortran = @XSDK_ENABLE_Fortran@
make.inc.in:ifeq ($(XSDK_ENABLE_Fortran),TRUE)
run_cmake_build.sh:#    -DXSDK_ENABLE_Fortran=TRUE \
run_cmake_build.sh:#    -DXSDK_ENABLE_Fortran=TRUE \

I tried adding the following change:

index adb53a740e..895a475b38 100644
--- a/var/spack/repos/builtin/packages/superlu-dist/package.py
+++ b/var/spack/repos/builtin/packages/superlu-dist/package.py
@@ -58,6 +58,7 @@ def cmake_args(self):
             '-DTPL_BLAS_LIBRARIES=%s' % spec['blas'].libs.joined(";"),
             '-DTPL_LAPACK_LIBRARIES=%s' % spec['lapack'].libs.joined(";"),
             '-DUSE_XSDK_DEFAULTS=YES',
+            '-DXSDK_ENABLE_Fortran=ON',
             '-DTPL_PARMETIS_LIBRARIES=%s' % spec['parmetis'].libs.ld_flags +
             ';' + spec['metis'].libs.ld_flags,
             '-DTPL_PARMETIS_INCLUDE_DIRS=%s' %

But get errors:

==> Installing superlu-dist-develop-yvx4hi6htcimmkm7vy3m6issuy2gpi2v
==> No binary for superlu-dist-develop-yvx4hi6htcimmkm7vy3m6issuy2gpi2v found: installing from source
==> Warning: Missing a source id for superlu-dist@develop
==> No patches needed for superlu-dist
==> superlu-dist: Executing phase: 'cmake'
==> superlu-dist: Executing phase: 'build'
==> superlu-dist: Executing phase: 'install'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j12' 'install'

1 error found in build log:
     932    -- Installing: /home/balay/spack/opt/spack/linux-fedora31-skylake/gcc-9.3.1/superlu-dist-develop-yvx4hi6htcimmkm7vy3m6issuy2gpi2v/include/superlu_FortranCInterface.h
     933    -- Installing: /home/balay/spack/opt/spack/linux-fedora31-skylake/gcc-9.3.1/superlu-dist-develop-yvx4hi6htcimmkm7vy3m6issuy2gpi2v/include/cub.cuh
     934    -- Installing: /home/balay/spack/opt/spack/linux-fedora31-skylake/gcc-9.3.1/superlu-dist-develop-yvx4hi6htcimmkm7vy3m6issuy2gpi2v/include/superlu_ddefs.h
     935    -- Installing: /home/balay/spack/opt/spack/linux-fedora31-skylake/gcc-9.3.1/superlu-dist-develop-yvx4hi6htcimmkm7vy3m6issuy2gpi2v/include/dlustruct_gpu.h
     936    -- Installing: /home/balay/spack/opt/spack/linux-fedora31-skylake/gcc-9.3.1/superlu-dist-develop-yvx4hi6htcimmkm7vy3m6issuy2gpi2v/lib/libsuperlu_dist_fortran.so
     937    -- Set runtime path of "/home/balay/spack/opt/spack/linux-fedora31-skylake/gcc-9.3.1/superlu-dist-develop-yvx4hi6htcimmkm7vy3m6issuy2gpi2v/lib/libsuperlu_dist_fortran.so" to "/home/balay/spack/opt/s
            pack/linux-fedora31-skylake/gcc-9.3.1/superlu-dist-develop-yvx4hi6htcimmkm7vy3m6issuy2gpi2v/lib"
  >> 938    CMake Error at FORTRAN/cmake_install.cmake:72 (file):
     939      file INSTALL cannot find
     940      "/home/balay/spack/opt/spack/linux-fedora31-skylake/gcc-9.3.1/superlu-dist-develop-yvx4hi6htcimmkm7vy3m6issuy2gpi2v/FORTRAN/superlu_mod.mod":
     941      No such file or directory.
     942    Call Stack (most recent call first):
     943      cmake_install.cmake:57 (include)
     944    

See build log for details:
  /home/balay/spack/spack-stage/spack-stage-superlu-dist-develop-yvx4hi6htcimmkm7vy3m6issuy2gpi2v/spack-build-out.txt

[balay]

And if I remove that single if statement:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 1c82490..ab6833d 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -393,14 +393,10 @@ endif()
 # Fortran-C name mangling 
 #
 ######################################################################
-if (XSDK_ENABLE_Fortran)
-  include(FortranCInterface)
-  FortranCInterface_HEADER(${SuperLU_DIST_SOURCE_DIR}/SRC/superlu_FortranCInterface.h
-  MACRO_NAMESPACE "FC_")
-  FortranCInterface_VERIFY(CXX)
-  SET(MPI_Fortran_LINK_FLAGS "${CMAKE_EXE_LINKER_FLAGS}")
-endif()
-
+include(FortranCInterface)
+FortranCInterface_HEADER(${SuperLU_DIST_SOURCE_DIR}/SRC/superlu_FortranCInterface.h
+MACRO_NAMESPACE "FC_")
+FortranCInterface_VERIFY(CXX)
 ######################################################################
 #
 # Include directories

I get:

==> Installing superlu-dist-develop-crhlsnfu5xfqd3av7ft4kvlguhtzxzlh
==> No binary for superlu-dist-develop-crhlsnfu5xfqd3av7ft4kvlguhtzxzlh found: installing from source
==> Warning: Missing a source id for superlu-dist@develop
==> Applied patch /home/balay/spack/var/spack/repos/builtin/packages/superlu-dist/fm.patch
==> superlu-dist: Executing phase: 'cmake'
==> Error: ProcessError: Command exited with status 1:
    'cmake' '-G' 'Unix Makefiles' '-DCMAKE_INSTALL_PREFIX:STRING=/home/balay/spack/opt/spack/linux-fedora31-skylake/gcc-9.3.1/superlu-dist-develop-crhlsnfu5xfqd3av7ft4kvlguhtzxzlh' '-DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo' '-DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF' '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=OFF' '-DCMAKE_INSTALL_RPATH:STRING=/home/balay/spack/opt/spack/linux-fedora31-skylake/gcc-9.3.1/superlu-dist-develop-crhlsnfu5xfqd3av7ft4kvlguhtzxzlh/lib;/home/balay/spack/opt/spack/linux-fedora31-skylake/gcc-9.3.1/superlu-dist-develop-crhlsnfu5xfqd3av7ft4kvlguhtzxzlh/lib64;/home/balay/spack/opt/spack/linux-fedora31-skylake/gcc-9.3.1/metis-5.1.0-f3rphiytdwf5eylh4mawn2qpdn5dj4wd/lib;/home/petsc/soft/mpich-3.3b1/lib;/home/balay/spack/opt/spack/linux-fedora31-skylake/gcc-9.3.1/parmetis-4.0.3-vv72c5r3vxcugvl6v7yvbdfreiczae7r/lib' '-DCMAKE_PREFIX_PATH:STRING=/home/petsc/soft/mpich-3.3b1;/home/balay/spack/opt/spack/linux-fedora31-skylake/gcc-9.3.1/parmetis-4.0.3-vv72c5r3vxcugvl6v7yvbdfreiczae7r;/home/balay/spack/opt/spack/linux-fedora31-skylake/gcc-9.3.1/metis-5.1.0-f3rphiytdwf5eylh4mawn2qpdn5dj4wd' '-DCMAKE_C_FLAGS=-std=c99' '-DCMAKE_CXX_FLAGS=-std=c++11' '-DCMAKE_C_COMPILER=/home/petsc/soft/mpich-3.3b1/bin/mpicc' '-DCMAKE_CXX_COMPILER=/home/petsc/soft/mpich-3.3b1/bin/mpic++' '-DCMAKE_INSTALL_LIBDIR:STRING=/home/balay/spack/opt/spack/linux-fedora31-skylake/gcc-9.3.1/superlu-dist-develop-crhlsnfu5xfqd3av7ft4kvlguhtzxzlh/lib' '-DTPL_BLAS_LIBRARIES=/usr/lib64/libblas.so' '-DTPL_LAPACK_LIBRARIES=/usr/lib64/liblapack.so' '-DUSE_XSDK_DEFAULTS=YES' '-DTPL_PARMETIS_LIBRARIES=-L/home/balay/spack/opt/spack/linux-fedora31-skylake/gcc-9.3.1/parmetis-4.0.3-vv72c5r3vxcugvl6v7yvbdfreiczae7r/lib -lparmetis;-L/home/balay/spack/opt/spack/linux-fedora31-skylake/gcc-9.3.1/metis-5.1.0-f3rphiytdwf5eylh4mawn2qpdn5dj4wd/lib -lmetis' '-DTPL_PARMETIS_INCLUDE_DIRS=/home/balay/spack/opt/spack/linux-fedora31-skylake/gcc-9.3.1/parmetis-4.0.3-vv72c5r3vxcugvl6v7yvbdfreiczae7r/include;/home/balay/spack/opt/spack/linux-fedora31-skylake/gcc-9.3.1/metis-5.1.0-f3rphiytdwf5eylh4mawn2qpdn5dj4wd/include' '-DXSDK_INDEX_SIZE=32' '-Denable_openmp=OFF' '-DCMAKE_DISABLE_FIND_PACKAGE_OpenMP=ON' '-DBUILD_SHARED_LIBS:BOOL=ON' '/home/balay/spack/spack-stage/spack-stage-superlu-dist-develop-crhlsnfu5xfqd3av7ft4kvlguhtzxzlh/spack-src'

1 error found in build log:
     21    -- Found MPI: TRUE (found version "3.1")
     22    -- Using TPL_BLAS_LIBRARIES='/usr/lib64/libblas.so'
     23    -- Will not link with LAPACK.
     24    -- Enabled support for PARMETIS.
     25    -- EXTRA_LIB_EXPORT='-lgcc_s -lgcc -lc -lgcc_s -lgcc -lmpi '
     26    -- Will not link with CombBLAS.
  >> 27    CMake Error at /usr/share/cmake/Modules/FortranCInterface.cmake:205 (message):
     28      FortranCInterface requires the Fortran language to be enabled.
     29    Call Stack (most recent call first):
     30      CMakeLists.txt:396 (include)
     31    
     32    
     33    -- Configuring incomplete, errors occurred!

See build log for details:
  /home/balay/spack/spack-stage/spack-stage-superlu-dist-develop-crhlsnfu5xfqd3av7ft4kvlguhtzxzlh/spack-build-out.txt

[balay]

I think its good to use the following as default code [so that the build works with majority of compilers without DXSDK_ENABLE_Fortran set]

#define FC_GLOBAL(name,NAME) name##_
#define FC_GLOBAL_(name,NAME) name##_

[and not commit any changes to this file to git. cmake appears to modify this file - so when compilers are changed in your build - looks like changes to this file get committed]

And I think we should fix spack to always use -DXSDK_ENABLE_Fortran=ON [and fix the issues that break here when this flag is set]. Some packages have fortran set as variant - but not all. [I'm not sure if we can use a fortran variant here - due to this dependency]

[drew-parsons]

I've had some related trouble with the fortran wrapper library. As discussed here, I've had to set -DXSDK_ENABLE_Fortran=ON in order to get the BLAS symbols handled with the underscore suffix.

But XSDK_ENABLE_Fortran=ON also triggers a build of libsuperlu_dist_fortran.so (which is probably a good idea. My build wasn't doing that previously). Some observations on libsuperlu_dist_fortran.so:

• it isn't linked to libsuperlu_dist.so
• the VERSION from toplevel CMakeFiles.txt isn't being propagated down to the superlu_dist_fortran target, so it isn't built with a versioned SONAME.
• the fortran header files are defined with CMAKE_INSTALL_PREFIX paths, which is problematic since it means the build gets immediately placed in /usr/include/ at build time. This is a problem for packaging (e.g. making deb or rpm packages), where we want the built targets to be installed to a destination dir for final packaging. CMAKE_BINARY_DIR works for me, rather than CMAKE_INSTALL_PREFIX

This patch addresses these points (assuming we want the same SOVERSION used for libsuperlu_dist.so)

Index: superlu-dist/FORTRAN/CMakeLists.txt
===================================================================
--- superlu-dist.orig/FORTRAN/CMakeLists.txt    2021-05-22 10:14:23.087104639 +0200
+++ superlu-dist/FORTRAN/CMakeLists.txt 2021-05-22 10:33:22.859494879 +0200
@@ -3,9 +3,9 @@
 include_directories(${SuperLU_DIST_BINARY_DIR}/FORTRAN)

 set(headers
-    ${CMAKE_INSTALL_PREFIX}/FORTRAN/superlu_mod.mod
-    ${CMAKE_INSTALL_PREFIX}/FORTRAN/superlupara_mod.mod
-    ${CMAKE_INSTALL_PREFIX}/FORTRAN/superlu_dist_config.fh
+    ${CMAKE_BINARY_DIR}/FORTRAN/superlu_mod.mod
+    ${CMAKE_BINARY_DIR}/FORTRAN/superlupara_mod.mod
+    ${CMAKE_BINARY_DIR}/FORTRAN/superlu_dist_config.fh
 #    ${CMAKE_CURRENT_BINARY_DIR}/fortran_config.h
     )

@@ -20,6 +20,11 @@

 add_library(superlu_dist_fortran ${sources})
 set(targets superlu_dist_fortran)
+get_target_property(superlu_dist_version superlu_dist VERSION)
+get_target_property(superlu_dist_soversion superlu_dist SOVERSION)
+set_target_properties(superlu_dist_fortran PROPERTIES VERSION ${superlu_dist_version})
+set_target_properties(superlu_dist_fortran PROPERTIES SOVERSION ${superlu_dist_soversion})
+target_link_libraries(superlu_dist_fortran superlu_dist)

 # depends on FPP defs
 add_dependencies(superlu_dist_fortran config_f)

[drew-parsons]

There's also a problem with the build of the test fortran programs if a parallel build is used (e.g. make -j8). I haven't been able to resolve this problem. The test programs (f_pddrive, f_5x5, etc) use superlupara.f90 and superlu_mod.f90, which build to superlupara_mod.mod and superlu_mod.mod respectively. But these fortran mod files apparently need to be compiled before f_*.o object files are compiled, otherwise with a make -j8 parallel build there usually is an error like

[ 95%] Building Fortran object FORTRAN/CMakeFiles/f_pddrive3d.dir/superlupara.f90.o
cd /build/superlu-dist/obj-x86_64-linux-gnu/FORTRAN && /usr/bin/gfortran  -I/usr/lib/x86_64-linux-gnu/fortran/gfortran-mod-15/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/lib -I/build/superlu-dist/obj-x86_64-linux-gnu/SRC -I/usr/include/CombBLAS -I/build/superlu-dist/SRC -I/build/superlu-dist/obj-x86_64-linux-gnu/FORTRAN -pthread -g -O2 -ffile-prefix-map=/build/superlu-dist=. -fstack-protector-strong -fPIE   -pthread -c /build/superlu-dist/FORTRAN/superlupara.f90 -o CMakeFiles/f_pddrive3d.dir/superlupara.f90.o
f951: Fatal Error: Cannot rename module file ‘superlupara_mod.mod0’ to ‘superlupara_mod.mod’: No such file or directory
compilation terminated.
make[3]: *** [FORTRAN/CMakeFiles/f_pddrive3d.dir/build.make:85: FORTRAN/CMakeFiles/f_pddrive3d.dir/superlupara.f90.o] Error 1

A sequential (single process) build works fine. It looks as if a build dependency between f_pddrive.o and superlupara_mod.mod , superlu_mod.mod needs to be declared, but isn't being handled automatically by cmake's FortranCInterface. I've tried some explicit dependency declarations with statements like

set_property(SOURCE f_pddrive.F90 PROPERTY OBJECT_DEPENDS ${F_MOD})

but I haven't found the right statement that makes a parallel build work reliably yet.

[xiaoyeli]

@balay In CMakeLists.txt, I set XSDK_ENABLE_Fortran=ON by default. I added /SRC/superlu_FortranCInterface.h in .gitignore file. Does this ensure that this file is not committed in git ? (I think I still need this file to be in git, otherwise when I make a release, this file would be missing ?

[drew-parsons]

There's also a problem with the build of the test fortran programs if a parallel build is used (e.g. make -j8).

The fortran parallel build problem is reported elsewhere, in netcdf-fortran #64, #208 , PR#238. Their issue#64 recommends using cmake object libraries. PR#238 deals with it using automake tools (using .lo).

https://stackoverflow.com/questions/29728867/cmake-parallel-build-for-fortran-90-with-modules gives an example of using module object libraries in cmake without automake.

[drew-parsons]

Following the hints in those links, this patch to FORTRAN/CMakeLists.txt makes a parallel build work:

@@ -52,37 +57,37 @@
 endif ()
 
 
-set(F_MOD superlupara.f90 superlu_mod.f90)
+add_library(ftestmod STATIC superlupara.f90 superlu_mod.f90)
 
 if(enable_double)
   set(F_DEXM ${F_MOD} f_pddrive.F90)
   add_executable(f_pddrive ${F_DEXM})
-  target_link_libraries(f_pddrive ${all_link_libs})
+  target_link_libraries(f_pddrive ftestmod ${all_link_libs})
 #  set_target_properties(f_pddrive PROPERTIES LINKER_LANGUAGE Fortran)
   set_target_properties(f_pddrive PROPERTIES LINKER_LANGUAGE CXX LINK_FLAGS "${MPI_Fortran_LINK_FLAGS}")
   
-  set(F_DEXM3D ${F_MOD} f_pddrive3d.F90)
+  set(F_DEXM3D f_pddrive3d.F90)
   add_executable(f_pddrive3d ${F_DEXM3D})
-  target_link_libraries(f_pddrive3d ${all_link_libs})
+  target_link_libraries(f_pddrive3d ftestmod ${all_link_libs})
   set_target_properties(f_pddrive3d PROPERTIES LINKER_LANGUAGE CXX LINK_FLAGS "${MPI_Fortran_LINK_FLAGS}")
   
-  set(F_5x5 ${F_MOD} f_5x5.F90 sp_ienv.c)
+  set(F_5x5 f_5x5.F90 sp_ienv.c)
   add_executable(f_5x5 ${F_5x5})
-  target_link_libraries(f_5x5 ${all_link_libs})
+  target_link_libraries(f_5x5 ftestmod ${all_link_libs})
   set_target_properties(f_5x5 PROPERTIES LINKER_LANGUAGE CXX LINK_FLAGS "${MPI_Fortran_LINK_FLAGS}")
   
 endif()
 
 if(enable_complex16)
-  set(F_ZEXM ${F_MOD} f_pzdrive.F90)
+  set(F_ZEXM f_pzdrive.F90)
   add_executable(f_pzdrive ${F_ZEXM})
-  target_link_libraries(f_pzdrive ${all_link_libs})
+  target_link_libraries(f_pzdrive ftestmod ${all_link_libs})
 #  set_target_properties(f_pzdrive PROPERTIES LINKER_LANGUAGE Fortran)
   set_target_properties(f_pzdrive PROPERTIES LINKER_LANGUAGE CXX LINK_FLAGS "${MPI_Fortran_LINK_FLAGS}")
 
-  set(F_ZEXM3D ${F_MOD} f_pzdrive3d.F90)
+  set(F_ZEXM3D f_pzdrive3d.F90)
   add_executable(f_pzdrive3d ${F_ZEXM3D})
-  target_link_libraries(f_pzdrive3d ${all_link_libs})
+  target_link_libraries(f_pzdrive3d ftestmod ${all_link_libs})
   set_target_properties(f_pzdrive3d PROPERTIES LINKER_LANGUAGE CXX LINK_FLAGS "${MPI_Fortran_LINK_FLAGS}")
   
 endif()

Reading add_library docs, https://cmake.org/cmake/help/latest/command/add_library.html#object-libraries , an object library with add_library( ... OBJECT ...) is an alternative to add_library( ... STATIC ...), in which case I think it needs add_executable(f_pddrive ${F_DEXM} $<TARGET_OBJECTS:ftestmod>) instead of target_link_libraries(f_pddrive ftestmod ${all_link_libs}). I tested the STATIC variant, not the OBJECT variant. The difference is whether an actual libftestmod.a file is generated in the build dir or not (it doesn't affect installation either way).

[balay]

@xiaoyeli, @drew-parsons I've attempted a couple of builds of latest superlu-dist@develop and get failures With the latest superlu-dist@develop - I get: spack-build-out.txt

Using the above patch fm.patch.txt I get: spack-build-out2.txt

However - I'm now able to build with XSDK_ENABLE_Fortran=OFF

[drew-parsons]

I wonder if the success of my patch might possibly be predicated on some of the other patches that I'm applying, though I was expecting it to be independent from the others. The full set of patches I'm applying are collected at https://salsa.debian.org/science-team/superlu-dist/-/tree/experimental/debian/patches . None of the other patches apply directly to the FORTRAN subdir, though. I haven't pulled git master, I'm patching 7.0.0 at this point. The actual fortran patch I applied is fortran_shared_lib_soname.patch, you might need the "${CMAKE_INSTALL_PREFIX}→ ${CMAKE_BINARY_DIR}" part of it.

Does your build succeed if you force a single processor build? (switching off -j12 for make). I guess yours will succeed in that case.

[balay]

Thanks! I also needed the following bits from fortran_shared_lib_soname.patch

-    ${CMAKE_INSTALL_PREFIX}/FORTRAN/superlu_mod.mod
-    ${CMAKE_INSTALL_PREFIX}/FORTRAN/superlupara_mod.mod
-    ${CMAKE_INSTALL_PREFIX}/FORTRAN/superlu_dist_config.fh
+    ${CMAKE_BINARY_DIR}/FORTRAN/superlu_mod.mod
+    ${CMAKE_BINARY_DIR}/FORTRAN/superlupara_mod.mod
+    ${CMAKE_BINARY_DIR}/FORTRAN/superlu_dist_config.fh

Attached the patch that worked [wrt master branch]

fortran_shared_lib_soname-master.patch.txt

[xiaoyeli]

@drew-parsons @balay I applied your latest patch to master, all the issues related to FORTRAN/ wrapper should be resolved. Thanks!

[jacobrking]

@drew-parsons @balay @xiaoyeli Hi, I have a related issue with this part of the build system when using the fortran interface.

Right now there is a dependency chain for the libaries: superlu_dist_fortran (depends on) -> ftestmod -> superlu_dist. However ftestmod is not installed (the associated module files are installed in the include directory). Without the calls in ftestmod I am unable to link (and I suspect this would also be the case without lazy linking).

I think the right solution is to roll ftestmod into superlu_dist_fortran. I've revised the CMakeLists.txt to do this in the attached patch file.

Thanks!

superlu_dist_fortran.patch.txt

[drew-parsons]

Hi @jacobrking , in my build it's the fortran test executables which depend on ftestmod, not superlu_dist_fortran. For me superlu_dist_fortran has no direct references to ftestmod. I'm still patching 7.0.0, I'm not pulling git master yet.

[jacobrking]

Hi @drew-parsons, after I re-examine the dependencies you are correct: ftestmod -> superlu_dist_fortran -> superlu_dist.

I'd like to use the interface for the set_superlu_options that is available in superlu_mod.F90 which permits optional arguments. It seems like there is no disadvantage to installing this in superlu_dist_fortran? There's also no reason to install the .mod files without installing the associated calls in the library.

The patch is relatively simple as it just removes ftestmod and moves superlu_mod.f90 and superlupara.f90 into the ${sources} variable where they will be built into the superlu_dist_fortran library.

[drew-parsons]

Making set_superlu_options available in the fortran wrapper library sounds reasonable to me. Perhaps there were reasons for keeping it separate.

[xiaoyeli]

I agree with @jacobrking 's logic: superluparam.f90 and superlu_mod.f90 should belong to the SOURCES and compiled into the superlu_dist_fortran.a library. f_pddfrive.f90 etc. acts like user's program, it can simply be linked with superlu_dist_fortran.a.

[xiaoyeli]

@jacobrking While looking at the this thread, I remember that the change by @drew-parsons enables parallel build "make -j 4" to work in the FORTRAN/ wrapper. Can you please try "make -j 4" with you patch?

[jacobrking]

Sorry for the delay. I just tested and the patch works with "-j 4" on Cori.

[MarDiehl]

@xiaoyeli: Would it be possible to make a maintenance release that includes the fix?