Undefined symbol when building OLTP with shared libraries on Mac OS

[wjones127]

Describe your environment

When building from the master branch on MacOS, if BUILD_SHARED_LIBS=ON and either WITH_OLTP_HTTP=ON or WITH_OLTP_GRPC are on then we get build errors:

[1/16] Linking CXX shared library ext/src/http/client/curl/libopentelemetry_http_client_curl.dylib
FAILED: ext/src/http/client/curl/libopentelemetry_http_client_curl.dylib 
: && /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -O3 -DNDEBUG -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -dynamiclib -Wl,-headerpad_max_install_names  -o ext/src/http/client/curl/libopentelemetry_http_client_curl.dylib -install_name @rpath/libopentelemetry_http_client_curl.dylib ext/src/http/client/curl/CMakeFiles/opentelemetry_http_client_curl.dir/http_client_factory_curl.cc.o ext/src/http/client/curl/CMakeFiles/opentelemetry_http_client_curl.dir/http_client_curl.cc.o ext/src/http/client/curl/CMakeFiles/opentelemetry_http_client_curl.dir/http_operation_curl.cc.o  /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/lib/libcurl.tbd && :
Undefined symbols for architecture arm64:
  "opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel()", referenced from:
      opentelemetry::v1::ext::http::client::curl::HttpOperation::SetCurlPtrOption(CURLoption, void*) in http_operation_curl.cc.o
      opentelemetry::v1::ext::http::client::curl::HttpOperation::SetCurlLongOption(CURLoption, long) in http_operation_curl.cc.o
      opentelemetry::v1::ext::http::client::curl::HttpOperation::Setup() in http_operation_curl.cc.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[2/16] Linking CXX shared library exporters/ostream/libopentelemetry_exporter_ostream_metrics.dylib
FAILED: exporters/ostream/libopentelemetry_exporter_ostream_metrics.dylib 
: && /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -O3 -DNDEBUG -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -dynamiclib -Wl,-headerpad_max_install_names  -o exporters/ostream/libopentelemetry_exporter_ostream_metrics.dylib -install_name @rpath/libopentelemetry_exporter_ostream_metrics.dylib exporters/ostream/CMakeFiles/opentelemetry_exporter_ostream_metrics.dir/src/metric_exporter.cc.o  -Wl,-rpath,/Users/willjones/Documents/opentelemetry-cpp/build/sdk/src/metrics -Wl,-rpath,/Users/willjones/Documents/opentelemetry-cpp/build/sdk/src/common  sdk/src/metrics/libopentelemetry_metrics.dylib  sdk/src/common/libopentelemetry_common.dylib && :
Undefined symbols for architecture arm64:
  "opentelemetry::v1::sdk::resource::Resource::GetAttributes() const", referenced from:
      opentelemetry::v1::exporter::metrics::OStreamMetricExporter::printResources(opentelemetry::v1::sdk::resource::Resource const&) in metric_exporter.cc.o
ld: symbol(s) not found for architecture arm64

Steps to reproduce

git clone [email protected]:open-telemetry/opentelemetry-cpp.git
cd opentelemetry-cpp

mkdir build
pushd build

mamba create -y -n otel-build \
       --channel=conda-forge
       clang_osx-arm64=14 \
       clang-tools=14 \
       libprotobuf=3.19 \
       python=3.9

cmake ${CMAKE_ARGS} ..  \
          -GNinja \
          -DCMAKE_BUILD_TYPE=Release \
          -DBUILD_SHARED_LIBS=ON \
          -DCMAKE_CXX_FLAGS="$CXXFLAGS" \
          -DCMAKE_CXX_STANDARD=17 \
          -DCMAKE_PREFIX_PATH=$CONDA_PREFIX \
          -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX \
          -DBUILD_TESTING=OFF \
          -DOPENTELEMETRY_INSTALL=ON \
          -DWITH_API_ONLY=OFF \
          -DWITH_BENCHMARK=OFF \
          -DWITH_EXAMPLES=OFF \
          -DWITH_LOGS_PREVIEW=ON \
          -DWITH_OTLP=ON \
          -DWITH_OTLP_HTTP=OFF \
          -DWITH_OTLP_GRPC=ON \
          -DWITH_PROMETHEUS=ON \
          -DWITH_ZIPKIN=ON

ninja install

What is the expected behavior?

I expected the build to create shared libraries without error.

What is the actual behavior?

Got the linking errors described above.

Additional context

I discovered this issue while trying to setup conda-forge to distribute shared libraries.

https://github.com/conda-forge/cpp-opentelemetry-sdk-feedstock/pull/41

[owent]

We need curl for any HTTP exporter by now. Do you have available libcurl of arm64 in your environment? Could you please provide what shows when cmake try to find curl?

[wjones127]

I don't think this has to do with not finding CURL. But good point, I should share my CMake configure log.

CMake Configure Log

-- The C compiler identification is AppleClang 13.1.6.13160021

-- The CXX compiler identification is AppleClang 13.1.6.13160021
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Building for architecture ARCH=arm64
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
Trying to use local prometheus-cpp from submodule
-- Performing Test HAVE_CXX_ATOMICS_WITHOUT_LIB
-- Performing Test HAVE_CXX_ATOMICS_WITHOUT_LIB - Success
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success
-- Found ZLIB: /Users/willjones/mambaforge/lib/libz.dylib (found version "1.2.13") 
-- Found CURL: /Users/willjones/mambaforge/lib/libcurl.dylib (found version "7.87.0")  
-- The following features have been enabled:

 * Pull, support for pulling metrics
 * Push, support for pushing metrics to a push-gateway
 * Compression, support for zlib compression of metrics
 * pkg-config, generate pkg-config files

-- The following OPTIONAL packages have been found:

 * Threads

-- The following REQUIRED packages have been found:

 * civetweb-3rdparty
 * ZLIB
 * CURL

-- The following features have been disabled:

 * IYWU, include-what-you-use

-- Found Protobuf: /Users/willjones/mambaforge/lib/libprotobuf.dylib (found version "3.19.4") 
CMake Warning at CMakeLists.txt:336 (find_package):
  By not providing "FindgRPC.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "gRPC", but
  CMake did not find one.

  Could not find a package configuration file provided by "gRPC" with any of
  the following names:

    gRPCConfig.cmake
    grpc-config.cmake

  Add the installation prefix of "gRPC" to CMAKE_PREFIX_PATH or set
  "gRPC_DIR" to a directory containing one of the above files.  If "gRPC"
  provides a separate development package or SDK, be sure it has been
  installed.


CMake Warning at CMakeLists.txt:356 (find_package):
  By not providing "FindgRPC.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "gRPC", but
  CMake did not find one.

  Could not find a package configuration file provided by "gRPC" with any of
  the following names:

    gRPCConfig.cmake
    grpc-config.cmake

  Add the installation prefix of "gRPC" to CMAKE_PREFIX_PATH or set
  "gRPC_DIR" to a directory containing one of the above files.  If "gRPC"
  provides a separate development package or SDK, be sure it has been
  installed.


-- PROTOBUF_PROTOC_EXECUTABLE=/Users/willjones/mambaforge/bin/protoc
-- opentelemetry-proto dependency satisfied by: git submodule
-- Performing Test check_cxx_compiler_flag_-Wno-type-limits
-- Performing Test check_cxx_compiler_flag_-Wno-type-limits - Success
-- Performing Test check_cxx_compiler_flag_-Wno-deprecated-declarations
-- Performing Test check_cxx_compiler_flag_-Wno-deprecated-declarations - Success
-- Performing Test check_cxx_compiler_flag_-Wno-unused-parameter
-- Performing Test check_cxx_compiler_flag_-Wno-unused-parameter - Success
-- Found CURL: /Users/willjones/mambaforge/lib/libcurl.dylib, version 
-- nlohmann::json dependency satisfied by: package
Building with nostd types...
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/willjones/Documents/opentelemetry-cpp/build

[owent]

HttpClient will not be built if we find headers of libcurl but got a wrong target architecture of the library. I'm not sure but add -DCMAKE_OSX_ARCHITECTURES=arm64 -DCMAKE_SYSTEM_PROCESSOR=aarch64 when run cmake may help.

I have not tested building to arm64 with default toolchains on macOS, but I have tested use ios toolchains to build into arm64 architecture (The ios.test job in https://github.com/atframework/cmake-toolset/actions) . With this toolset, we also set CMAKE_OSX_SYSROOT, CMAKE_SYSROOT. And also we build all dependencies and add -DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER -DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY -DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY -DCMAKE_FIND_ROOT_PATH_MODE_PACKAGE=ONLY or cmake may find local prebuilt without arm64 in find_package()/find_libraries() which will also lead to failure.

[wjones127]

I found that if I make these changes, I could successfully link on MacOS:

https://github.com/wjones127/opentelemetry-cpp/commit/83b8395396e680725e04613b9d157e815df79223

Perhaps there are some differences in how the MacOS linker works? I'll look a little more into this.

[owent]

I found that if I make these changes, I could successfully link on MacOS:

wjones127@83b8395

Perhaps there are some differences in how the MacOS linker works? I'll look a little more into this.

Does "CMake test (build shared libraries with otlp-exporter and static gRPC)" (https://github.com/open-telemetry/opentelemetry-cpp/blob/main/.github/workflows/ci.yml#L369) works with these changes on macOS? On ELF platform, we can not link static gRPC into more than one dynamic libraries or exectuable.

[Korevishal]

I am seeing the same error on windows, was able to produce all 17 shared libs produced by OpenTelemetry for HTTP but still this dependency is not resolved NvOpenTracing.lib(NvOpenTelemetry.obj) : error LNK2019: unresolved external symbol "private: static struct std::pair<class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::sdk::common::internal_log::LogHandler>,enum opentelemetry::v1::sdk::common::internal_log::LogLevel> & __cdecl opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::GetHandlerAndLevel(void)" (?GetHandlerAndLevel@GlobalLogHandler@internal_log@common@sdk@v1@opentelemetry@@CAAEAU?$pair@V?$shared_ptr@VLogHandler@internal_log@common@sdk@v1@opentelemetry@@@nostd@v1@opentelemetry@@W4LogLevel@internal_log@common@sdk@34@@std@@XZ) referenced in function "public: static void __cdecl opentelemetry::v1::sdk::common::internal_log::GlobalLogHandler::SetLogHandler(class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::sdk::common::internal_log::LogHandler>)" (?SetLogHandler@GlobalLogHandler@internal_log@common@sdk@v1@opentelemetry@@SAXV?$shared_ptr@VLogHandler@internal_log@common@sdk@v1@opentelemetry@@@nostd@56@@Z)

[lalitb]

@wjones127 Do you have further update on the issue.

[Korevishal]

I tried the @wjones127 method but it didn't work. Apparently, my issue was I tried to turn on STL with cmake option DWITH_STL=ON but had not defined macro HAVE_CPP_STDLIB This doc says you need, to make option or macro to enable STL but actually you need to define cmake option and macro both. with DWITH_STL=OFF problem is resolved for me The doc needs to be changed to reflect it more appropriately. Thanks

[github-actions[bot]]

This issue was marked as stale due to lack of activity.