OpenColorIO
OpenColorIO copied to clipboard
AcademySoftwareFoundation
centos7 static build - lib missing symbols from pystring, expat, cpp-yaml
Hello ! I am having problems with statically build OCIO. When I try to use it in (static) build of OIIO I get lots of undefined references for: pystring, expat, cpp-yaml:
/v/pkgs/ocio/install_gcc_6.1_static/lib/libOpenColorIO.a(Context.cpp.o): In function OpenColorIO_v2_0::Context::resolveFileLocation(char const*, std::shared_ptr<OpenColorIO_v2_0::Context>&) const:
Context.cpp:(.text+0x32f6): undefined reference to pystring::os::path::isabs(std::string const&)
/v/pkgs/ocio/install_gcc_6.1_static/lib/libOpenColorIO.a(OCIOYaml.cpp.o): In function OpenColorIO_v2_0::(anonymous namespace)::saveDescription(YAML::Emitter&, char const*):
OCIOYaml.cpp:(.text+0xb9): undefined reference to YAML::Emitter::SetLocalValue(YAML::EMITTER_MANIP)
/v/pkgs/ocio/install_gcc_6.1_static/lib/libOpenColorIO.a(FileFormatCTF.cpp.o): In function OpenColorIO_v2_0::(anonymous namespace)::LocalFileFormat::read(std::istream&, std::string const&, OpenColorIO_v2_0::Interpolation) const:
FileFormatCTF.cpp:(.text+0x6c82): undefined reference to XML_ParserCreate
Indeed, when I examine the ocio lib, I see lots of U undefined symbols, for example:
nm ./libOpenColorIO.a | grep pystring | c++filt
U pystring::os::path::join(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
nm ./libOpenColorIO.a | grep XML | c++filt
U XML_ErrorString
nm ./libOpenColorIO.a | grep YAML | c++filt
U typeinfo for YAML::InvalidNode
Here is cmake options and I attach make build.log and cmake cache.
cmake
-DBUILD_SHARED_LIBS=OFF
-DCMAKE_INSTALL_PREFIX=/v/pkgs/ocio/install_gcc_6.1_static
-DOCIO_BUILD_TESTS=OFF
-DOCIO_BUILD_PYTHON=OFF
-DOCIO_BUILD_JAVA=OFF
-DOCIO_INSTALL_EXT_PACKAGES=MISSING
-DHalf_STATIC_LIBRARY=ON
-Dpystring_STATIC_LIBRARY=ON
-Dpystring_ROOT=/v/pkgs/pystring/2021-08-06/pystring
-Dpystring_INCLUDE_DIR=/v/pkgs/pystring/2021-08-06/pystring
-Dexpat_STATIC_LIBRARY=ON
-Dexpat_ROOT=/v/pkgs/expat/install_static
-Dyaml-cpp_ROOT=/v/pkgs/yaml-cpp/install_static
-Dyaml-cpp_STATIC_LIBRARY=ON
../OpenColorIO
-- The CXX compiler identification is GNU 6.3.1 -- The C compiler identification is GNU 6.3.1 -- Check for working CXX compiler: /opt/rh/devtoolset-6/root/usr/bin/c++ -- Check for working CXX compiler: /opt/rh/devtoolset-6/root/usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Check for working C compiler: /opt/rh/devtoolset-6/root/usr/bin/cc -- Check for working C compiler: /opt/rh/devtoolset-6/root/usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Setting build type to 'Release' as none was specified. -- Setting C++ version to '11' as none was specified. -- Performing Test COMPILER_SUPPORTS_CXX11 -- Performing Test COMPILER_SUPPORTS_CXX11 - Success -- Found OpenGL: /usr/lib64/libOpenGL.so found components: OpenGL -- Found GLEW: /usr/include -- Found GLUT: /usr/lib64/libglut.so -- GLVND supported -- Performing Test HAVE_SSE2 -- Performing Test HAVE_SSE2 - Success -- Setting SOVERSION to '2.0' as none was specified. -- Found expat: /v/pkgs/expat/install_static/lib64/libexpat.a (found suitable version "2.4.1", minimum required is "2.2.8") -- Found yaml-cpp: /v/pkgs/yaml-cpp/install_static/lib64/libyaml-cpp.a (Required is at least version "0.6.3") -- Could NOT find Half (missing: Half_LIBRARY Half_INCLUDE_DIR) (Required is at least version "2.4.0") -- Installing Half (IlmBase): /v/pkgs/ocio/build_static/ext/dist/lib64/libHalf-2_4.a (version "2.4.0") -- Found pystring: /v/pkgs/pystring/2021-08-06/pystring (Required is at least version "1.1.3") -- Could NOT find lcms2 (missing: lcms2_INCLUDE_DIR lcms2_LIBRARY) (Required is at least version "2.2") -- Installing lcms2: /v/pkgs/ocio/build_static/ext/dist/lib/liblcms2.a (version "2.2") -- Could NOT find OpenImageIO (missing: OPENIMAGEIO_INCLUDE_DIR OPENIMAGEIO_LIBRARY OPENIMAGEIO_VERSION) (Required is at least version "2.1.9") -- Use "OpenImageIO_ROOT" to specify an install location CMake Warning at src/apps/CMakeLists.txt:17 (message): Could NOT find OpenImageIO. Skipping build for ociolutimage, ocioconvert, ociodisplay and ocioperf
-- Configuring done -- Generating done -- Build files have been written to: /v/pkgs/ocio/build_static
make VERBOSE=1
@hodoulp Yes, the logs are for compilation of OCIO - which seems to go OK. Troubles arise when I try to link OCIO into OIIO (which is also statically compiled).
Below are my OIIO cmake options. I also attach CMake files and make logs for OIIO compilation.
export OpenColorIO_ROOT=/v/pkgs/ocio/install_gcc_6.3.1_static export OPENEXR_ROOT=//bbcfg/pkgs/prod/openexr/2.2.0/linux64_static export BOOST_ROOT=/v/pkgs/boost/dep/1_61_0 export LIBS="/v/pkgs/expat/install_static/lib64/libexpat.a /usr/lib/libpystring.a /v/pkgs/yaml-cpp/install_static/lib64/libyaml-cpp.a"
cmake
-DCMAKE_INSTALL_PREFIX=/v/pkgs/oiio/devel/2021-08-05/install_gcc-6.3.1_static
-DUSE_PYTHON=0
-DOIIO_BUILD_TESTS=0
-DBUILD_SHARED_LIBS=OFF
../oiio
make BUILD_SHARED_LIBS=0 LINKSTATIC=1 VERBOSE=1
OIIO_cmake_init.log OIIO_CMakeCache.txt OIIO_build.log
The third-party libraries of OpenColorIO were supposed to be embedded by default even when compiling a static library so, there is definitively a problem.
On MacOs with Ninja 1.10.1, cmake 3.20.1 and gcc 12.0.0:
~/dev/ocio_priv_2/build_rls (master) $ cmake -GNinja -DBUILD_SHARED_LIBS=OFF ../.
-- The CXX compiler identification is AppleClang 12.0.0.12000032
-- The C compiler identification is AppleClang 12.0.0.12000032
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Setting build type to 'Release' as none was specified.
-- Setting C++ version to '11' as none was specified.
-- Performing Test COMPILER_SUPPORTS_CXX11
-- Performing Test COMPILER_SUPPORTS_CXX11 - Success
-- Found OpenGL: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/System/Library/Frameworks/OpenGL.framework found components: OpenGL
-- Found GLUT: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/System/Library/Frameworks/GLUT.framework
-- GLVND not supported; legacy OpenGL libraries used
-- Performing Test HAVE_SSE2
-- Performing Test HAVE_SSE2 - Success
-- Setting SOVERSION to '2.1' as none was specified.
-- Found expat: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/usr/lib/libexpat.tbd (found suitable version "2.2.8", minimum required is "2.2.8")
-- Could NOT find yaml-cpp (missing: yaml-cpp_LIBRARY yaml-cpp_INCLUDE_DIR) (Required is at least version "0.6.3")
CMake Warning at share/cmake/modules/Findyaml-cpp.cmake:201 (message):
Building STATIC libOpenColorIO using the in-built yaml-cpp. yaml-cpp
symbols are NOT included in the output binary!
Call Stack (most recent call first):
share/cmake/modules/FindExtPackages.cmake:33 (find_package)
CMakeLists.txt:241 (include)
-- Installing yaml-cpp: /Users/hodoulp/dev/ocio_priv_2/build_rls/ext/dist/lib/libyaml-cpp.a (version "0.6.3")
-- Could NOT find Half (missing: Half_LIBRARY Half_INCLUDE_DIR) (Required is at least version "2.4.0")
-- Installing Half (IlmBase): /Users/hodoulp/dev/ocio_priv_2/build_rls/ext/dist/lib/libHalf-2_4.a (version "2.4.0")
-- Could NOT find pystring (missing: pystring_INCLUDE_DIR pystring_LIBRARY) (Required is at least version "1.1.3")
CMake Warning at share/cmake/modules/Findpystring.cmake:122 (message):
Building STATIC libOpenColorIO using the in-built pystring. pystring
symbols are NOT included in the output binary!
Call Stack (most recent call first):
share/cmake/modules/FindExtPackages.cmake:41 (find_package)
CMakeLists.txt:241 (include)
-- Installing pystring: /Users/hodoulp/dev/ocio_priv_2/build_rls/ext/dist/lib/libpystring.a (version "1.1.3")
-- Found lcms2: /usr/local/Cellar/little-cms2/2.12/include (found suitable version "2.12", minimum required is "2.2")
-- Found Python: /usr/local/Frameworks/Python.framework/Versions/3.9/bin/python3.9 (found version "3.9.5") found components: Interpreter Development Development.Module Development.Embed
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Performing Test HAS_FLTO_THIN
-- Performing Test HAS_FLTO_THIN - Success
-- Found pybind11: /usr/local/include (found suitable version "2.6.2", minimum required is "2.6.1")
-- Found OpenImageIO: /usr/local/include (found suitable version "2.2.14.0", minimum required is "2.1.9")
-- OpenImageIO includes = /usr/local/include
-- OpenImageIO libraries = /usr/local/lib/libOpenImageIO.dylib
-- OpenImageIO library_dirs = /usr/local/lib
-- OpenImageIO oiiotool = /usr/local/bin/oiiotool
CMake Warning at src/bindings/python/CMakeLists.txt:44 (message):
Building PyOpenColorIO with OCIO_BUILD_DOCS disabled will result in
incomplete Python docstrings.
-- Configuring done
-- Generating done
Then I compiled and checked only one method from expat (no warning like yaml and pystring)
~/dev/ocio_priv_2/build_rls (master) $ find ../src -name "*.cpp" | xargs grep XML_ParserCreate
../src/OpenColorIO/fileformats/cdl/CDLParser.cpp: : m_parser(XML_ParserCreate(NULL))
../src/OpenColorIO/fileformats/FileFormatIridasLook.cpp: : m_parser(XML_ParserCreate(NULL))
../src/OpenColorIO/fileformats/FileFormatCTF.cpp: : m_parser(XML_ParserCreate(nullptr))
The symbol is still unknown
~/dev/ocio_priv_2/build_rls (master) $ nm ./src/OpenColorIO/libOpenColorIO.a | grep XML_ParserCreate | c++filt
U _XML_ParserCreate
U _XML_ParserCreate
U _XML_ParserCreate
Yes, I did provide my own build of Expat, yaml-cpp and pystring, to avoid these warnings: "Building STATIC libOpenColorIO using the in-built pystring. pystring symbols are NOT included in the output binary!"
Hi, any insights into this issue? Unclear if this is related as I'm building shared, but I seem to be crashing on a similar problem:
libOpenColorIO.so: undefined symbol: _ZN4YAML4LoadERNSt3__113basic_istreamIcNS0_11char_traitsIcEEEE
Doing the readelf -aW ... | c++filt | grep move on the library indicates that the symbol is undefined:
GLOBAL DEFAULT UND YAML::Load(std::__1::basic_istream<char, std::__1::char_traits<char> >&)
In libyaml-cpp.a:
24 FUNC LOCAL DEFAULT 27 YAML::Load(std::basic_istream<char, std::char_traits<char> >&) [clone .cold]
226 FUNC GLOBAL HIDDEN 7 YAML::Load(std::basic_istream<char, std::char_traits<char> >&)
(OCIO v2.0.1)
Edit: Ok, the issue seems to arise in my case if I try to force libc++ instead of libstdc++, which fails to propagate to OCIO's dependencies causing the missing symbols issue.
Hi @num3ric ,
Not sure if that's related, could probably go into a dedicated issue with the step to reproduce ?
The third-party libraries of OpenColorIO were supposed to be embedded by default even when compiling a static library so, there is definitively a problem.
Sorry for not following, do you mean we embed the third party static libraries into our own static library generation ? How do we do that ?
Hello @marza-sergey, we have recently made some enhancements to the build system on the main branch (following the release of version 2.2.1). Could you please verify if you are still having the same issues with the main branch of OCIO?
I tested today with 6bf2dc0 under RockyLinux 8.7 and g++ 9.2.1 This time I let cmake download all dependencies. The problem persists, I cannot statically link OIIO to OCIO.
Here is an exact cmake command:
cmake \ -DCMAKE_BUILD_TYPE=Release \ -DBUILD_SHARED_LIBS=Off \ -DCMAKE_INSTALL_PREFIX=./install_static \ -DOCIO_INSTALL_EXT_PACKAGES=ALL \ -DOCIO_USE_OIIO_FOR_APPS=Off \ -DOCIO_BUILD_PYTHON=Off \ ../OpenColorIO-2.2.1
I also used make BUILD_SHARED_LIBS=0 LINKSTATIC=1 but I'm not sure if it makes a difference.
has anybody build a static OCIO ?
I was able to eventually build OIIO statically with these flags:
CMAKE_POSITION_INDEPENDENT_CODE=ON
BUILD_SHARED_LIBS=OFF
EMBEDPLUGINS=0
PYTHON_VERSION=3.7 # Also works for 3.9
USE_PYTHON=1
OIIO_BUILD_TESTS=0
ENABLE_OpenColorIO=1
ENABLE_LibRaw=1
ENABLE_OpenVDB=1
Boost_USE_STATIC_LIBS=1
Boost_ROOT=$BOOST_ROOT
Boost_NO_WARN_NEW_VERSIONS=1
LINKSTATIC=1
INSTALL_FONTS=1
ZLIB_LIBRARY=$PATH_TO_ZLIB_A
ZLIB_INCLUDE_DIR=$ZLIB_INCLUDE
ZLIB_USE_STATIC_LIBS=1
after building OCIO with these flags:
CMAKE_POSITION_INDEPENDENT_CODE=ON
BUILD_SHARED_LIBS=OFF
OCIO_BUILD_APPS=ON
OCIO_BUILD_PYTHON=ON
OCIO_BUILD_DOCS=OFF
OCIO_BUILD_TESTS=OFF
OCIO_BUILD_GPU_TESTS=OFF
OCIO_BUILD_JAVA=OFF
lcms2_USE_STASTIC_LIBS=ON
CMAKE_PREFIX_PATH=$CMAKE_PREFIX
pystring_INCLUDE_DIR=$PYSTRING_INCLUDE
Python_EXECUTABLE=$PATH_TO_PYTHON
I'm not sure which exact flags solved my issues.
I'm also building statically zilb, imath, openexr, libjpeg-turbo, jasper, libraw, xz, c-blosc, onetbb, openvdb, pybind11, yaml-cpp, minizip-ng, libexpat, pystring(I had to patch this one), little-cms, fmt, libtiff, libpng and robin-map.
I'm using centos 7 and Red Hat Developer Toolset 11
Thanks. None of these options helped. I suppose your success is building all dependencies by yourself. I'm not sure I have steam to go through this...
Ok I tried again with building all dependencies myself (and doing static builds everywhere), but still my libOpenColorIO.a has all those symbols missing. @marza-sergey if you do nm libOpenColorIO.a - do you see Undefined symbols for XML, YAML and others ?
It looks like only OCIO binaries are statically linked, but not the lib
Here's my cmake config
cmake
-DCMAKE_BUILD_TYPE=Release
-DBUILD_SHARED_LIBS=Off
-DCMAKE_POSITION_INDEPENDENT_CODE=ON
-DLINKSTATIC=1
-DBoost_USE_STATIC_LIBS=1
-DCMAKE_INSTALL_PREFIX=./install_static
-DOCIO_INSTALL_EXT_PACKAGES=NONE
-DImath_DIR=/bbcfg_local/pkgs/prod/Imath/3.1.9_19355e6/linux64_Rocky8_gcc_9.2.1_static/lib64/cmake/Imath
-DOpenEXR_DIR=/bbcfg_local/pkgs/prod/OpenEXR/3.1.8/linux64_Rocky8_gcc9.2.1_static/lib64/cmake/OpenEXR
-Dexpat_STATIC_LIBRARY=On
-Dexpat_DIR=/root/devel/expat/install_static/lib64/cmake/expat-2.5.0
-Dyaml-cpp_DIR=/root/devel/yaml-cpp/install_static/share/cmake/yaml-cpp
-Dpystring_ROOT=/root/devel/pystring/install_static
-DZLIB_ROOT=/root/devel/zlib/install_static
-Dminizip-ng_STATIC_LIBRARY=On
-Dminizip-ng_ROOT=/root/devel/minizip-ng/install_static_ocio
-Dlcms2_USE_STASTIC_LIBS=On
-Dlcms2_ROOT=/root/devel/lcms/install_static
-DOCIO_USE_OIIO_FOR_APPS=Off
-DOCIO_BUILD_PYTHON=Off
../OpenColorIO-2.2.1
Yes, I still have those symbols undefined in libOpenColorIO.a
$ nm libOpenColorIO.a | grep XML_ErrorString
U XML_ErrorString
U XML_ErrorString
U XML_ErrorString
I guess, this is how it supposed to work and all those symbols should be resolved from other libraries, when you compile binaries.
Then I should also link all those dependencied in OIIO project ? I will ask there. Thanks
I don't link them explicitly, but I have all the libraries installed in the same place and the path is included to CMAKE_PREFIX_PATH, so I think cmake/gcc can find and link them automatically.
I've run into this same issue myself with OCIO 2.x, and it's basically the second coming of this old issue: https://github.com/AcademySoftwareFoundation/OpenColorIO/issues/414
The current approach works when building a shared library, but there is no such thing as "linking" one static library to another on Linux, so the static OCIO library is left with missing dependency symbols.
The previous fix for this was to update the build process to build the embedded dependencies as CMake Object Library targets and then "link" the static OCIO library to those, which resulted in the compiled object files being embedded in the static OCIO library (see https://github.com/AcademySoftwareFoundation/OpenColorIO/pull/453).
However, the build process has regressed at some point to just use STATIC CMake targets again, so it's re-broken.
