trilinos
Framework: Unit tests fail to link with Intel 19 and shared libraries
Bug Report
@trilinos/framework @bartlettroscoe
When building Trilinos for EMPIRE with Intel 19 compilers and shared libraries, I encounter linking errors in various unit test binaries, with Panzer coming up first:
[18/654] Linking CXX executable packages/panzer/dof-mgr/test/fe_assembly/PanzerDofMgr_Test_FE_Assembly_HEX.exe
FAILED: packages/panzer/dof-mgr/test/fe_assembly/PanzerDofMgr_Test_FE_Assembly_HEX.exe
: && /projects/sems/install/rhel7-x86_64/sems/compiler/intel/19.0.5/openmpi/1.10.1/bin/mpicxx -D_GLIBCXX_USE_CXX11_ABI=0 -O3 -DNDEBUG -mkl packages/panzer/dof-mgr/test/fe_assembly/CMakeFiles/PanzerDofMgr_Test_FE_Assembly_HEX.dir/test_fe_assembly_HEX.cpp.o packages/panzer/dof-mgr/test/fe_assembly/CMakeFiles/PanzerDofMgr_Test_FE_Assembly_HEX.dir/__/cartesian_topology/CartesianConnManager.cpp.o -o packages/panzer/dof-mgr/test/fe_assembly/PanzerDofMgr_Test_FE_Assembly_HEX.exe -Wl,-rpath,/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/panzer/dof-mgr/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/intrepid2/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/shards/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/phalanx/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/sacado/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/panzer/core/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/tpetra/core/ext:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/tpetra/core/inout:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/tpetra/core/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/tpetra/core/compat:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/tpetra/tsqr/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/kokkos-kernels/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/kokkos/algorithms/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/kokkos/containers/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/epetra/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/teuchos/numerics/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/teuchos/remainder/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/teuchos/kokkoscomm/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/teuchos/comm/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/teuchos/kokkoscompat/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/teuchos/parameterlist/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/teuchos/parser/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/teuchos/core/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/kokkos/core/src packages/panzer/dof-mgr/src/libpanzer-dof-mgr.so.13.5 packages/intrepid2/src/libintrepid2.so.13.5 packages/shards/src/libshards.so.13.5 packages/phalanx/src/libphalanx.so.13.5 packages/sacado/src/libsacado.so.13.5 packages/panzer/core/src/libpanzer-core.so.13.5 packages/tpetra/core/ext/libtpetraext.so.13.5 packages/tpetra/core/inout/libtpetrainout.so.13.5 packages/tpetra/core/src/libtpetra.so.13.5 packages/tpetra/core/compat/libtpetraclassic.so.13.5 packages/tpetra/tsqr/src/libkokkostsqr.so.13.5 packages/kokkos-kernels/src/libkokkoskernels.so.13.5 packages/kokkos/algorithms/src/libkokkosalgorithms.so.13.5 packages/kokkos/containers/src/libkokkoscontainers.so.13.5 packages/epetra/src/libepetra.so.13.5 packages/teuchos/numerics/src/libteuchosnumerics.so.13.5 packages/teuchos/remainder/src/libteuchosremainder.so.13.5 packages/teuchos/kokkoscomm/src/libteuchoskokkoscomm.so.13.5 packages/teuchos/comm/src/libteuchoscomm.so.13.5 packages/teuchos/kokkoscompat/src/libteuchoskokkoscompat.so.13.5 packages/teuchos/parameterlist/src/libteuchosparameterlist.so.13.5 packages/teuchos/parser/src/libteuchosparser.so.13.5 packages/teuchos/core/src/libteuchoscore.so.13.5 packages/kokkos/core/src/libkokkoscore.so.13.5 -ldl && :
ld: packages/panzer/dof-mgr/test/fe_assembly/CMakeFiles/PanzerDofMgr_Test_FE_Assembly_HEX.dir/test_fe_assembly_HEX.cpp.o(.text+0xc38c): unresolvable R_X86_64_TPOFF32 relocation against symbol `_ZN6Kokkos4Impl22SharedAllocationRecordIvvE18t_tracking_enabledE'
ld: final link failed: Nonrepresentable section on output
This looks exactly like kokkos/kokkos#4722, and potentially has the same cause - setting CMAKE_POSITION_INDEPENDENT_CODE where it's not needed, since BUILD_SHARED_LIBRARIES is sufficient.
With this patch, the relevant executable builds successfully:
diff --git a/cmake/std/atdm/ATDMDevEnvSettings.cmake b/cmake/std/atdm/ATDMDevEnvSettings.cmake
index c5a6c1c..f35157d 100644
--- a/cmake/std/atdm/ATDMDevEnvSettings.cmake
+++ b/cmake/std/atdm/ATDMDevEnvSettings.cmake
@@ -217,10 +217,6 @@ IF ((ATDM_COMPILER MATCHES ".*CLANG.*") AND ATDM_ADDRESS_SANITIZER)
SET(EXTRA_EXTRA_LINK_FLAGS
"${EXTRA_EXTRA_LINK_FLAGS} -ldl -fsanitize=address")
ENDIF()
-# FPIC
-IF (ATDM_FPIC)
- SET(CMAKE_POSITION_INDEPENDENT_CODE ON CACHE BOOL "Build targets with position independent code")
-ENDIF()
SET(CMAKE_C_FLAGS
"$ENV{ATDM_CONFIG_C_FLAGS} ${CMAKE_C_FLAGS}")
I also see it getting set in these places, but these changes or commenting them out made no difference in my case:
diff --git a/packages/framework/ini-files/config-specs.ini b/packages/framework/ini-files/config-specs.ini
index 0fd0f53..eda3f53 100644
--- a/packages/framework/ini-files/config-specs.ini
+++ b/packages/framework/ini-files/config-specs.ini
@@ -670,7 +670,7 @@ opt-set-cmake-var Trilinos_ENABLE_COMPLEX BOOL : OFF
#
[USE-FPIC|YES]
-opt-set-cmake-var CMAKE_POSITION_INDEPENDENT_CODE BOOL : ON
+opt-set-cmake-var CMAKE_POSITION_INDEPENDENT_CODE BOOL : OFF
[USE-FPIC|NO]
# Nothing to do here.
diff --git a/packages/framework/pr_tools/trilinosprhelpers/unittests/config-specs.ini b/packages/framework/pr_tools/trilinosprhelpers/unittests/config-specs.ini
index 6174dd7..583e480 100644
--- a/packages/framework/pr_tools/trilinosprhelpers/unittests/config-specs.ini
+++ b/packages/framework/pr_tools/trilinosprhelpers/unittests/config-specs.ini
@@ -606,7 +606,7 @@ opt-set-cmake-var Trilinos_ENABLE_COMPLEX BOOL : OFF
#
[USE-FPIC|YES]
-opt-set-cmake-var CMAKE_POSITION_INDEPENDENT_CODE BOOL : ON
+opt-set-cmake-var CMAKE_POSITION_INDEPENDENT_CODE BOOL : OFF
[USE-FPIC|NO]
# Nothing to do here.
Steps to Reproduce
- SHA1: 29305794b0b1e9b03ea75d0af4597a2df4de50b4
+ Ran the following on ascic169:
build_trilinos.py \
--compiler intel-19 \
--build-type opt \
--node-type serial \
--lib-type shared \
--src-dir /ascldap/users/pbmille/repos \
--ref develop \
--save-replay-file /scratch/pbmille/trilinos/.build_trilinos_replay_2022-09-01_18.24.51.014436 \
--build-system ninja \
--stage configure
+ Trilinos configured in:
/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED
with:
cmake \
-D Trilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake,cmake/std/atdm/apps/empire/EMPIRETrilinosEnables.cmake \
-D Panzer_ENABLE_EXAMPLES:BOOL=ON \
-D Panzer_ENABLE_TESTS:BOOL=ON \
-D TPL_ENABLE_gtest=OFF \
-D Trilinos_ENABLE_Tempus=OFF \
-D CMAKE_CXX_COMPILER_LAUNCHER=$(which ccache) \
-D CMAKE_C_COMPILER_LAUNCHER=$(which ccache) \
-D CMAKE_INSTALL_PREFIX=/scratch/pbmille/trilinos/install/INTEL-19.0.5-RELEASE-SERIAL-SHARED \
-D TPL_ENABLE_CGNS=ON \
-D CGNS_LIBRARY_DIRS=${CGNS_ROOT}/lib \
-D CGNS_INCLUDE_DIRS=${CGNS_ROOT}/include \
-D TPL_ENABLE_CUSPARSE:BOOL=OFF \
-G Ninja \
/home/pbmille/repos/Trilinos
Thanks for the workaround @PhilMiller ! I've filed a ticket for this with the framework team so they are aware of it - TRILINOSHD-199.
Hello @PhilMiller, if you post a PR for the updates to files under cmake/std/atdm/ I will approve.
https://github.com/trilinos/Trilinos/pull/10984 @bartlettroscoe There's the PR
This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.