Visibility Issues Guidance

[gregcotten]

I realize this isn't a C++ support forum, but I was wondering if you had any guidance on compiling with "hidden" visibility of dependencies.

For example, I build OIIO + dependencies completely statically, but want all the dependencies of OIIO to be hidden so that (for example) OIIO's boost dependency doesn't get used for things out of scope of the OIIO libraries. Is this possible to do? Any special incantation for boost b2 compile I need to do?

A warning I see a million times during OIIO compilation (different symbols, same warning):

ld: warning: direct access in function 'boost::exception_ptr boost::exception_detail::get_static_exception_object<boost::exception_detail::bad_alloc_>()' from file '{redacted}/boost-1.79.0/lib/libboost_thread-mt.a(thread.o)' to global weak symbol 'guard variable for boost::exception_ptr boost::exception_detail::get_static_exception_object<boost::exception_detail::bad_alloc_>()::ep' from file '../../lib/libOpenImageIO_Util.a(sysutil.cpp.o)' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.

Additionally I will see some warnings like this:

{redacted}/oiio-2.3.16.0/src/include/OpenImageIO/benchmark.h:29:1: warning: 'always_inline' attribute ignored [-Wignored-attributes]
OIIO_FORCEINLINE T const& DoNotOptimize (T const &val);
^
{redacted}/oiio-2.3.16.0/src/include/OpenImageIO/platform.h:393:52: note: expanded from macro 'OIIO_FORCEINLINE'
#    define OIIO_FORCEINLINE inline __attribute__((always_inline))
                                                   ^
{redacted}/oiio-2.3.16.0/src/include/OpenImageIO/benchmark.h:431:33: note: conflicting attribute is here
inline T const & __attribute__((__optnone__))

Thanks!

[lgritz]

No worries, Greg, I'm actually always looking for ways to improve the build and dependency experience. I don't consider myself an expert in a lot of this area, and am trying to learn myself.

Your second example related to the always_inline/optnone conflict, I believe, should be fixed by #3444. It's in both master as well as the dev-2.3 staging branches already, and I expect it to be in the next 2.3.17.0 release scheduled for July 1.

I haven't seen the first warning before, the one about boost. I'm not really sure I understand it myself. So you're saying that this is a warning you're seeing when building YOUR app, by linking OIIO? Or when you are building OIIO itself?

We do try to hide symbols. One way we do it is here and using this file and some extra logic here. The idea is that we are defaulting our own symbols to be hidden except if they are part of the public APIs, and also trying to suppress the visibility of symbols of any of our dependencies.

Though I see now that some parts of this logic are specific to Linux. So maybe that's why it's not working for you as expected.

Which platform are you using, and which compilers?

[gregcotten]

Thanks for replying so soon!

I'm seeing the "This was likely caused by different translation units being compiled with different visibility settings. " warning during OIIO compilation (they also show up when compiling my app when linking OIIO, too, though!).

I'm targeting macOS universal - I wonder if that dependent library hiding would work out of the box on macOS - I'll give it a go!

[lgritz]

I have no recollection now about why that dependent library hiding seems crafted only for Linux (and FreeBSD and GNU) but excludes MacOS. I do a lot of my development on a Mac laptop, so it's not like I didn't probably try it. I suspect you're about to find out why it's like that. :-)

[gregcotten]

OK, so apologies: these warnings were only coming up during the executable (like iconvert) link phase at the very end of olio compilation.

Adding "Darwin" to the list of supported CMAKE_SYSTEM_NAME sets the variable correctly and compiles just fine - it doesn't seem to affect these warnings though.

I did make some progress, however. I use CMake for building OIIO, so I added

-DCMAKE_CXX_FLAGS="-fvisibility=hidden -fvisibility-inlines-hidden"

and that completely cleared up any boost visibility warnings. However, I still am getting OpenEXR (2.5.x - macOS universal2 build support on 3.x is still iffy) warnings:

/ld:1:1: direct access in function 'OpenImageIO_v2_3::OpenEXRInput::PartInfo::parse_header(OpenImageIO_v2_3::OpenEXRInput*, Imf_2_5::Header const*)' from file '../Vendor/lib/OpenImageIO//libOpenImageIO.a(exrinput.cpp.o)' to global weak symbol 'typeinfo for Imf_2_5::TypedAttribute<Imath_2_5::Matrix33<float> >' from file '../Vendor/lib/OpenImageIO//libOpenImageIO.a(exroutput.cpp.o)' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.

/ld:1:1: direct access in function 'Imf_2_5::Header::hasName() const' from file '../Vendor/lib/openexr//libIlmImf-2_5.a(ImfHeader.cpp.o)' to global weak symbol 'typeinfo for Imf_2_5::TypedAttribute<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >' from file '../Vendor/lib/OpenImageIO//libOpenImageIO.a(exroutput.cpp.o)' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.

[gregcotten]

The boost::asio dependency is the main culprit for me. I have a plugin that shares memory space with other plugins that use different versions of asio which causes terrible crashes (Why isn't boost ABI stable?!). Turns out there is a hard to find compiler definition (BOOST_ASIO_DISABLE_VISIBILITY for hiding ASIO symbols as they don't respect the usual methods of hiding.

This is "good enough" for my current issue, though there are plenty of other boost symbols that get leaked (not sure what to do there!)

This would be my proposed patch (will do a PR at some point...)

if (${CXX_VISIBILITY_PRESET} STREQUAL "hidden" AND VISIBILITY_MAP_FILE AND
    (CMAKE_COMPILER_IS_GNUCC OR CMAKE_COMPILER_IS_CLANG) AND
    (CMAKE_SYSTEM_NAME MATCHES "Linux|kFreeBSD" OR CMAKE_SYSTEM_NAME STREQUAL "GNU" OR CMAKE_SYSTEM_NAME STREQUAL "Darwin"))
    # Linux/macOS/FreeBSD/Hurd: also hide all the symbols of dependent libraries
    # to prevent clashes if an app using this project is linked against
    # other versions of our dependencies.
    add_compile_definitions(BOOST_ASIO_DISABLE_VISIBILITY)
    set (VISIBILITY_MAP_COMMAND "-Wl,--version-script=${VISIBILITY_MAP_FILE}")
endif ()