rstanarm build error: Library not loaded: @rpath/libtbb.dylib

[barracuda156]

I am making an implementation for R packages in Macports now, and I would like to have rstanarm among supported packages. This is still WIP, however what mostly remains to be done is fine-tuning for Macports environment. That is, pretty much everything already builds fine – both locally on my machines and on Macports build bots.

I get an error, however, with rstanarm: for some reason, it cannot find libtbb.dylib, even though it is there (see log below). No external version of TBB is installed, so the failure is not caused by duplicate dylibs. Existing libtbb.dylib is sitting in RcppParallel package, and linker flag is pointing to the correct location: -L/opt/local/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppParallel/lib/ -ltbb -ltbbmalloc. Yet when testing package loading, it fails to find it.

:info:destroot /opt/local/bin/g++-mp-12 -std=gnu++14 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I"../inst/include" -I"/opt/local/Library/Frameworks/R.framework/Versions/4.2/Resources/library/StanHeaders/include/src" -DBOOST_DISABLE_ASSERTS -DEIGEN_NO_DEBUG -I'/opt/local/Library/Frameworks/R.framework/Versions/4.2/Resources/library/StanHeaders/include' -I'/opt/local/Library/Frameworks/R.framework/Versions/4.2/Resources/library/rstan/include' -I'/opt/local/Library/Frameworks/R.framework/Versions/4.2/Resources/library/BH/include' -I'/opt/local/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I'/opt/local/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include' -I'/opt/local/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppParallel/include' -I/opt/local/include  `"/opt/local/Library/Frameworks/R.framework/Resources/bin/Rscript" -e "RcppParallel::CxxFlags()"` `"/opt/local/Library/Frameworks/R.framework/Resources/bin/Rscript" -e "StanHeaders:::CxxFlags()"` -fPIC  -pipe -Os -arch ppc  -c init.cpp -o init.o
:info:destroot /opt/local/bin/g++-mp-12 -std=gnu++14 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/opt/local/Library/Frameworks/R.framework/Resources/lib -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-rpath,/opt/local/lib/libgcc -arch ppc -o rstanarm.so stan_files/bernoulli.o stan_files/binomial.o stan_files/continuous.o stan_files/count.o stan_files/jm.o stan_files/lm.o stan_files/mvmer.o stan_files/polr.o init.o -L/opt/local/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppParallel/lib/ -ltbb -ltbbmalloc -F/opt/local/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
:info:destroot rm stan_files/binomial.cc stan_files/jm.cc stan_files/continuous.cc stan_files/bernoulli.cc stan_files/lm.cc stan_files/polr.cc stan_files/count.cc stan_files/mvmer.cc
:info:destroot installing to /opt/local/var/macports/build/_opt_PPCRosettaPorts_R_R-rstanarm/R-rstanarm/work/destroot/opt/local/Library/Frameworks/R.framework/Versions/4.2/Resources/library/00LOCK-stan-dev-rstanarm-e63e48a/00new/rstanarm/libs
:info:destroot ** R
:info:destroot ** data
:info:destroot *** moving datasets to lazyload DB
:info:destroot ** demo
:info:destroot ** inst
:info:destroot ** byte-compile and prepare package for lazy loading
:info:destroot ** help
:info:destroot No man pages found in package  ‘rstanarm’ 
:info:destroot *** installing help indices
:info:destroot *** copying figures
:info:destroot ** building package indices
:info:destroot ** installing vignettes
:info:destroot ** testing if installed package can be loaded from temporary location
:info:destroot Error: package or namespace load failed for ‘rstanarm’ in dyn.load(file, DLLpath = DLLpath, ...):
:info:destroot  unable to load shared object '/opt/local/var/macports/build/_opt_PPCRosettaPorts_R_R-rstanarm/R-rstanarm/work/destroot/opt/local/Library/Frameworks/R.framework/Versions/4.2/Resources/library/00LOCK-stan-dev-rstanarm-e63e48a/00new/rstanarm/libs/rstanarm.so':
:info:destroot   dlopen(/opt/local/var/macports/build/_opt_PPCRosettaPorts_R_R-rstanarm/R-rstanarm/work/destroot/opt/local/Library/Frameworks/R.framework/Versions/4.2/Resources/library/00LOCK-stan-dev-rstanarm-e63e48a/00new/rstanarm/libs/rstanarm.so, 6): Library not loaded: @rpath/libtbb.dylib
:info:destroot   Referenced from: /opt/local/var/macports/build/_opt_PPCRosettaPorts_R_R-rstanarm/R-rstanarm/work/destroot/opt/local/Library/Frameworks/R.framework/Versions/4.2/Resources/library/00LOCK-stan-dev-rstanarm-e63e48a/00new/rstanarm/libs/rstanarm.so
:info:destroot   Reason: image not found
:info:destroot Error: loading failed
:info:destroot Execution halted
:info:destroot ERROR: loading failed

P. S. It is probably desirable to dump an old tbb in RcppParallel altogether and use an external oneTBB (given that we have fixed it for ppc32, so all macOS are covered), but I am not sure if that gonna work smoothly, since ABI has changed quite a bit.

@andrjohns @WardBrian Any ideas here?

[WardBrian]

Are you also setting the rpath to include that location? It seems like no, which would explain why it links but fails at runtime

[barracuda156]

Are you also setting the rpath to include that location? It seems like no, which would explain why it links but fails at runtime

@WardBrian I have tried to do that after I got the failure, but to no avail. It is quite possible though, that I made some mistake in the relevant chunk of code. I will look into that.

[barracuda156]

On a side-note, rstanarm appears to be the only package failing this way. Other dependents of RcppParallel link and load correctly. For example, bayesforecast:

/opt/local/bin/g++-mp-12 -std=gnu++14 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/opt/local/Library/Frameworks/R.framework/Resources/lib -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-rpath,/opt/local/lib/libgcc -arch ppc -o bayesforecast.so RcppExports.o stanExports_SVM.o stanExports_Sarima.o stanExports_ets.o stanExports_tgarch.o -L/opt/local/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppParallel/lib/ -Wl,-rpath,/opt/local/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppParallel/lib/ -ltbb -ltbbmalloc -F/opt/local/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
installing to /opt/local/var/macports/build/_opt_PPCRosettaPorts_R_R-bayesforecast/R-bayesforecast/work/destroot/opt/local/Library/Frameworks/R.framework/Versions/4.2/Resources/library/00LOCK-bayesforecast/00new/bayesforecast/libs
** R
** data
*** moving datasets to lazyload DB
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (bayesforecast)

[barracuda156]

@WardBrian This fixed the build:

--- src/Makevars.orig	2022-04-09 03:53:34.000000000 +0700
+++ src/Makevars	2022-12-19 06:37:41.000000000 +0700
@@ -4,7 +4,7 @@
 PKG_CXXFLAGS = `"${R_HOME}/bin$(R_ARCH_BIN)/Rscript" -e "RcppParallel::CxxFlags()"` `"${R_HOME}/bin$(R_ARCH_BIN)/Rscript" -e "StanHeaders:::CxxFlags()"`
 SHLIB_LDFLAGS = $(SHLIB_CXXLDFLAGS)
 SHLIB_LD = $(SHLIB_CXXLD)
-PKG_LIBS = `"${R_HOME}/bin$(R_ARCH_BIN)/Rscript" -e "RcppParallel::RcppParallelLibs()"` -L"$(TBB_LIB)" -ltbb -ltbbmalloc
+PKG_LIBS = `"${R_HOME}/bin$(R_ARCH_BIN)/Rscript" -e "RcppParallel::RcppParallelLibs()"` -L"$(TBB_LIB)" -Wl,-rpath,"$(TBB_LIB)" -ltbb -ltbbmalloc
 
 CXX_STD = CXX14
 SOURCES = $(wildcard stan_files/*.stan)

[andrjohns]

rstanarm TBB linking will be fixed with the merge of #574

[barracuda156]

rstanarm TBB linking will be fixed with the merge of #574

@andrjohns Sounds good. I hope the new arrangement will remain compatible with my Macports solution (the PR you have probably seen).

P. S. BTW, any plans of moving to the new oneTBB? I have it fixed for PPC (and likely i386): https://github.com/oneapi-src/oneTBB/pull/840 Old TBB builds fine with a few fixes (existing RcppParallel code does not work correctly), but I guess no one gonna work on it.