Installing from source fails on macOS

[pat-s]

Should be the same as #241 but with a more descriptive title.

I am facing this issue locally since months - in addition, the GitHub Actions install failures cause headaches from time to time. Time to put some weight onto it :)

I saw you moved the issue to the next milestone. Do you have any idea when it will be tackled?

Note that CRAN tests on macOS 10.13 and the latest SDKs are somewhat troublesome for source installs. Hence, while your package might install properly on CRAN, it will error on 90% of all user machines when doing source installs because they are > 10.13.

It does not need to be SDK releated necessarily. Maybe some brew linkings gets into the way. However, given that the minimal GitHub Actions builds also show the error, I am somewhat confident that it is not related to a local config problem of mine :)

The error below uses the default clang from macOS (on macOS 10.15), as it is the standard for R >= 4.0.

** building package indices
** testing if installed package can be loaded from temporary location
Error: package or namespace load failed for ‘fst’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/Users/pjs/Library/R/4.0/library/00LOCK-fst/00new/fst/libs/fst.so':
  dlopen(/Users/pjs/Library/R/4.0/library/00LOCK-fst/00new/fst/libs/fst.so, 6): Symbol not found: _XXH32
  Referenced from: /Users/pjs/Library/R/4.0/library/00LOCK-fst/00new/fst/libs/fst.so
  Expected in: flat namespace
 in /Users/pjs/Library/R/4.0/library/00LOCK-fst/00new/fst/libs/fst.so

[gabrimaine]

I solved the problem installing fst via with_makevars to set the path to llvm:

v<-c("-L/usr/local/opt/llvm/lib","-I/usr/local/opt/llvm/include")
names(v)=c("LDFLAGS","CPPFLAGS") 
with_makevars(v, install.packages("fst"), path = file.path("~", ".R", "Makevars"), c("="))``` 

[pat-s]

@gabrimaine I confirm that this works, thanks! Maybe this can help @MarcusKlik to implement a generic fix.

[MarcusKlik]

Hi @gabrimaine, thanks for sharing your solution and @pat-s thanks for testing!

I've activated the R-CMD-check action on the fstcore repository to reproduce your error, see here. We can use that to test a fix. The thing is, I can't just add the LDFLAGS and CPPFLAGS to the package Makevars, as users might not have installed the llvm compiler or use another install location...

[pat-s]

Looks like an OpenMP issue to me, i.e. the wrong pointers are set / or the wrong are found during source installation.

If I install the binary, OpenMP is also not detected (which is expected given that {data.table} also does not find it by default).

library(fst)
fst package v0.9.4
(OpenMP was not detected, using single threaded mode)

Not sure how {fst} does it but maybe looking how other popular packages (e.g. data.table) handle OpenMP might be fruitful?

See also https://mac.r-project.org/openmp/. Following these instructions I can install {data.table} with OpenMP support. In my case, I have OpenMp installed via brew install libomp. The important pieces are

CPPFLAGS= -I/usr/local/opt/libomp/include -Xclang -fopenmp
LDFLAGS += -lomp

Note that the first piece -I/usr/local/opt/libomp/include is specific to detect the local libomp install.

library(data.table)
data.table 1.13.0 using 4 threads (see ?getDTthreads).  Latest news: r-datatable.com

# fst still errors

[matthewgson]

@pat-s I also encountered this issue and I solved it with adding CXX11 = $(LLVM_LOC)/bin/g++ -fopenmp in Makevars file. For details, please refer to https://github.com/Rdatatable/data.table/wiki/Installation

[pat-s]

Thanks.

For many reasons this should be supported OOB in the package using Apples default clang compiler. The issue is still present with the new 11.0 SDK.

The reason that CRAN checks are not erroring probably relies on CRAN checking on macOS 10.13 and not against the most recent SDKs.

[MarcusKlik]

Hi @pat-s, you're definitely right, this should work OOB for macOS users. The data.table package is using a configure script now to detect OpenMP on macOS and I think it's a good idea to use a similar strategy for fst to avoid these installation problems.

[MarcusKlik]

Package fstcore is now tested on macOS with and without the bundled fstlib libraries. Please let me know if there are any more problems compiling the package on MacOS.