Error in conda forge build

[brendanf]

trafficstars

It looks like the conda-forge package for fst has not been able to successfully build on OSX since they upgraded their dependencies back in July, and there isn't any action there to diagnose the problem. The error given is the same one from https://github.com/fstpackage/fst/issues/61#issuecomment-441479592. It's not your responsibility to maintain the conda-forge build, but I wonder if you might have any insight? The recipe includes llvm-openmp as a dependency on OSX, which seems to work successfully for other R packages (e.g., influencer, mgcv, data.table).

The error message seems like it may be somehow related to XXHash32, not to OpenMP?

Error: package or namespace load failed for ‘fst’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '$PREFIX/lib/R/library/fst/libs/fst.so':
  dlopen($PREFIX/lib/R/library/fst/libs/fst.so, 6): Symbol not found: _XXH32
  Referenced from: $PREFIX/lib/R/library/fst/libs/fst.so
  Expected in: flat namespace
 in /usr/local/minico
Error: loading failed

[MarcusKlik]

Hi @brendanf, thanks for the heads-up!

Yes, I can see from the osx build output that llvm-openmp has been added to the build. But from the compiler output from fst (at the bottom) it seems that the llvm compiler is not used to build the package, but instead the default x86_64-apple-darwin13.4.0-clang++ compiler.

On Travis, I added the following exports to activate llvm clang:

export PATH="/usr/local/opt/llvm/bin:$PATH"
export LDFLAGS="-L/usr/local/opt/llvm/lib"
export CPPFLAGS="-I/usr/local/opt/llvm/include"

Those settings will ensure that the llvm clang compiler can be found and is used for package compilation.

The specific error message thrown in the conda forge build seems unrelated to OpenMP but is, as you say, a problem with linking to the LZ4 compression library. That library (and ZSTD) use .c files, where the rest of the code is written in C++. Somehow, the linking fails between code compiled with clang and that compiled with clang++.

Perhaps it's best to fix the llvm installation first and see from there :-)

Is there a way to recreate the conda build locally, so changes in the build script can be tested more easily?

thanks!

[brendanf]

Is there a way to recreate the conda build locally, so changes in the build script can be tested more easily?

You can build the package locally by:

• installing miniconda
• installing the build environment with conda install conda-build
• clone the feedstock repository
• conda build r-fst-feedstock/recipe

I'm using Linux, so this doesn't help me test the issue on OSX.

On Travis, I added the following exports to activate llvm clang:

export PATH="/usr/local/opt/llvm/bin:$PATH"
export LDFLAGS="-L/usr/local/opt/llvm/lib"
export CPPFLAGS="-I/usr/local/opt/llvm/include"

The conda build recipe shouldn't use anything in /usr/local; but presumably llvm is installed somewhere in the directory given by $CONDA_PREFIX.

[MarcusKlik]

Hi @brendanf,

I noticed that the install_name_tool in the recipe still uses the links to the 'old' R installation (see this file).

From other comments (e.g. here) I understand that the version of libc++ bundled by R might not be compatible with the OSX system libraries.

Therefore it might be that updating the links to the correct 3.6.0 location might help fix the build, what do you think, is that something that we can test?

thanks

[brendanf]

The build.sh for the recipe appears to be generated by conda skeleton cran, and most other R packages on Conda use exactly the same thing. I think the code referencing R 3.5 is unreachable, i.e.

if [[ ... ]] || [[ ... ]] || [[ ... ]] || [[ $target_platform == osx-64 ]]; then
  ...
else
  ...
  if [[ $target_platform == osx-64 ]]; then
    # unreachable code
  fi
fi

I assume it is left over from a previous incarnation of the build script. In any case, the exact same code is present in other recipes which build fine on OSX.

[brendanf]

I've been digging into this more and trying a few variations, but I haven't found anything that helps with the linker issues. Unfortunately I haven't been able to figure out how to cross-build on my own (linux) machine, so all my testing has been through conda-forge's Azure pipeline. I have learned:

• Conda-forge's clang is already the llvm version, not the Apple distributed version. (here). So the llvm-openmp package is just providing those specific libraries.
• I tried reverting all changes since the last successful conda-forge build, but it still failed. Unfortunately, the logs from the last successful build are no longer available. This implies that the issue is not changes to the recipe, or even the build environment as specified here but changes to the overall conda-forge ecosystem.
• Maybe this is some edge-case with a newer version of clang or clang++?

[MarcusKlik]

Hi @brendanf, the same here! I've tried a lot of different combinations of compiler flags and environmental variables (see these builds) but wasn't able to successfully compile the package.

What I did notice was that the compiler flags that I try to change by using CDFLAGS and CPPFLAGS exports (like in the fst travis setup) don't see to translate to the actual package build.

For example the clang compiler needs the following flags:

  export LDFLAGS="-L$BUILD_PREFIX/lib $LDFLAGS"
  export CPPFLAGS="-I$BUILD_PREFIX/include $CPPFLAGS"

in order for the build to use the correct libc++ libraries. However, the settings above don't translate to the flags being set during the build, so something strange is going on there...

[MarcusKlik]

So the problem boils down to:

• to successfully compile fst on OSX we need both clang and clang++ from llvm

• these compilers need to link to the correct libraries during builds as indicated by the output of the (homebrew) llvm installer:

Pouring llvm-9.0.0_1.high_sierra.bottle.tar.gz
==> Caveats

To use the bundled libc++ please add the following LDFLAGS:
  LDFLAGS="-L/usr/local/opt/llvm/lib -Wl,-rpath,/usr/local/opt/llvm/lib"

llvm is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have llvm first in your PATH run:
  echo 'export PATH="/usr/local/opt/llvm/bin:$PATH"' >> ~/.bash_profile

For compilers to find llvm you may need to set:
  export LDFLAGS="-L/usr/local/opt/llvm/lib"
  export CPPFLAGS="-I/usr/local/opt/llvm/include"

• The llvm libraries in the conda build seem to be installed in $BUILD_PREFIX/lib. This build output includes a command ls - R where that can be seen, the contents of that folder are:

$BUILD_PREFIX/lib:
clang
libLLVM-9.dylib
libLTO.9.dylib
libRemarks.9.dylib
libc++.1.0.dylib
libc++.1.dylib
libc++.a
libc++.dylib
libclang-cpp.9.dylib
libclang-cpp.dylib
libtapi.dylib
tapi

(so inlcuding libc++.a)

• My attempts to add -L$BUILD_PREFIX/lib to the build of the fst package have not be successful however. Without these libraries added to the compilers search path, we can expect linking problems like the one that we have now.

So I think the question really is what the correct way is to add -L$BUILD_PREFIX/lib to the fst build?

what do you think? Thanx for working on this!

[brendanf]

I'm not sure exactly how, but it looks like the conda-forge build process was updated in a way that fixed this problem. The package is building fine on all platforms now.

[MarcusKlik]

great, thanks a lot for reporting!

[brendanf]

It looks like I spoke too soon in https://github.com/fstpackage/fst/issues/226#issuecomment-604964764 -- I thought this was resolved because I was able to get the version I wanted in conda, but it turns out it was coming from the default r channel, not from conda-forge.

[MarcusKlik]

Hi @brendanf, thanks for the heads up, that's very unfortunate! Looks like we have to dig deeper into the exact conda build system setup to get to the bottom of this...