`ldd` and `libc++` is not compatible to `librocksdb`

[xis19]

When building foundationdb using clang compiler set together with ldd, ldd could not find any symbols from librocksdb.a. Switching back to ld and disabling libc++ would fix the issue.

[sfc-gh-mpilman]

Using ld will increase link times significantly, so I don't like the idea of using that.

When using clang and libc++, RocksDB and its C++ dependencies also has to be compiled with that configuration. Has this been done?

[sears]

I checked out current origin/main (4cbe8be45). rm -rf ~/build_output ; ccmk works for me in okteto.

That's the only clang configuration we test with. How are you building it?

ccmk expands to:

CC=clang CXX=clang++ cmake -S ${HOME}/src/foundationdb -B ${HOME}/build_output -D USE_CCACHE=ON -D USE_WERROR=ON -D RocksDB_ROOT=/opt/rocksdb-6.27.3 -D RUN_JUNIT_TESTS=ON -D RUN_JAVA_INTEGRATION_TESTS=ON -G Ninja && ninja -C ${HOME}/build_output -j 84

[sears]

Prepending USE_LD=LLD to that also works. (In cmake/ConfigureCompiler.cmake, the help string for USE_LD says that USE_LD=LLD is the default with clang, but the logic below doesn't seem to explicitly override the variable. Either way, that didn't reproduce the issue.)

[sfc-gh-mpilman]

The main difference is that we disable RocksDB. Here's our default cmake (it's pretty long mostly because it is generated, not everything here will have an effect):

cmake -GNinja -S /mnt/ephemeral/mpilman/foundationdb -B /mnt/ephemeral/mpilman/build/foundationdb.linux.clang.x86_64 -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DFORCE_BOOST_BUILD=ON -DCMAKE_BUILD_TYPE=Release -DUSE_GPERFTOOLS=OFF -DUSE_DTRACE=ON -DUSE_VALGRIND=OFF -DALLOC_INSTRUMENTATION=OFF -DUSE_ASAN=OFF -DUSE_GCOV=OFF -DUSE_MSAN=OFF -DUSE_TSAN=OFF -DUSE_UBSAN=OFF -DUSE_LTO=OFF -DFDB_RELEASE=OFF -DRELATIVE_DEBUG_PATHS=OFF -DUSE_WERROR=OFF -DSSD_ROCKSDB_EXPERIMENTAL=OFF -DTEST_PACKAGE_EXCLUDE=VersionVector

So it seems -DSSD_ROCKSDB_EXPERIMENTAL=OFF -DTEST_PACKAGE_EXCLUDE=VersionVector are the two differences.

Is /opt/rocksdb-6.27.3 a compiled RocksDB? Even if it is not, my guess is that it will be compiled with libstdc++ and NOT with libc++. And this will break stuff

[sfc-gh-mpilman]

a relatively simple way of testing my hypothesis would be to add -DUSE_LIBCXX=OFF. If you do this everything should use libstdc++ (though this configuration hasn't been used for a long time, so something else might break).

But I would be very very surprised if RocksDB would use libc++ unless you did something special

[sears]

I got it to reproduce the link error with the command:

rm -rf ~/build_output/ && CC=clang CXX=clang++ cmake -D USE_LIBCXX=ON -D USE_LD=LLD -S ${HOME}/src/foundationdb -B ${HOME}/build_output -D USE_CCACHE=ON -D USE_WERROR=ON -D RocksDB_ROOT=/opt/rocksdb-6.27.3 -D RUN_JUNIT_TESTS=ON -D RUN_JAVA_INTEGRATION_TESTS=ON -G Ninja && ninja -C ${HOME}/build_output -j 84

The interesting part is:

CC=clang CXX=clang++ cmake -D USE_LIBCXX=ON -D USE_LD=LLD

In particular, passing in USE_LIBCXX=ON switched it from working to broken. What's the desired behavior for the clang build? I assumed it [edit: the clang build] would use libc++ by default.

We re-build rocksdb each time. From a G++ CI log:

[182/1547] cd /codebuild/output/src923461041/src/github.com/apple/foundationdb/build_output/fdbserver/rocksdb-prefix/src/rocksdb-build && /usr/local/bin/cmake -DUSE_RTTI=1 -DPORTABLE=ON -DCMAKE_CXX_STANDARD=17 -DCMAKE_CXX_COMPILER=/opt/rh/devtoolset-8/root/usr/bin/c++ -DCMAKE_BUILD_TYPE=Release -DWITH_GFLAGS=OFF -DWITH_TESTS=OFF -DWITH_TOOLS=OFF -DWITH_CORE_TOOLS=OFF -DWITH_BENCHMARK_TOOLS=OFF -DWITH_BZ2=OFF -DWITH_LZ4=ON -DWITH_SNAPPY=OFF -DWITH_ZLIB=OFF -DWITH_ZSTD=OFF -DWITH_LIBURING=OFF -DWITH_TSAN=OFF -DWITH_ASAN=OFF -DWITH_UBSAN=OFF -DROCKSDB_BUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=True -GNinja /opt/rocksdb-6.27.3 && /usr/local/bin/cmake -E touch /codebuild/output/src923461041/src/github.com/apple/foundationdb/build_output/fdbserver/rocksdb-prefix/src/rocksdb-stamp/rocksdb-configure

then, later:

[280/312] ccache /opt/rh/devtoolset-8/root/usr/bin/c++ -DHAVE_PCLMUL -DHAVE_SSE42 -DLZ4 -DOS_LINUX -DROCKSDB_FALLOCATE_PRESENT -DROCKSDB_LIB_IO_POSIX -DROCKSDB_MALLOC_USABLE_SIZE -DROCKSDB_NO_DYNAMIC_EXTENSION -DROCKSDB_PLATFORM_POSIX -DROCKSDB_PTHREAD_ADAPTIVE_MUTEX -DROCKSDB_RANGESYNC_PRESENT -DROCKSDB_SCHED_GETCPU_PRESENT -DROCKSDB_SUPPORT_THREAD_LOCAL -I/opt/rocksdb-6.27.3 -I/opt/rocksdb-6.27.3/include -I/opt/rocksdb-6.27.3/third-party/folly -W -Wextra -Wall -pthread -Wsign-compare -Wshadow -Wno-unused-parameter -Wno-unused-variable -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -Wno-strict-aliasing -fno-omit-frame-pointer -momit-leaf-frame-pointer -Werror -fno-builtin-memcmp -O3 -DNDEBUG -DROCKSDB_USE_RTTI -fPIC -std=gnu++17 -MD -MT CMakeFiles/rocksdb.dir/utilities/write_batch_with_index/write_batch_with_index.cc.o -MF CMakeFiles/rocksdb.dir/utilities/write_batch_with_index/write_batch_with_index.cc.o.d -o CMakeFiles/rocksdb.dir/utilities/write_batch_with_index/write_batch_with_index.cc.o -c /opt/rocksdb-6.27.3/utilities/write_batch_with_index/write_batch_with_index.cc

[sfc-gh-mpilman]

The clang build uses libc++ by default. However, the RocksDB build also needs to be told to use this. So basically the logic that compiles RocksDB needs to change so that cmake passes the correct arguments. The cmake call to compile RocksDB is hard coded which won't work if you want to support this setup

[sears]

I just realized that this is set in my developer container:

[root@sears-dev-okteto-6ccd4bdf89-qvjcz foundationdb]# env | grep -i cxx
USE_LIBCXX=0

USE_LIBCXX doesn't appear in the container Dockerfile:

https://github.com/FoundationDB/fdb-build-support/blob/main/docker/centos7/Dockerfile

or in ~/.bash*, /etc/*. I'm grepping all of ~ now. Strange.

[sears]

I found the issue; it's specific to our infrastructure.

[sears]

Pull request is here:

https://github.com/apple/foundationdb/pull/8265

[xis19]

Fixed by passing CMAKE_CXX_FLAGS to RocksDB build