Updates to prepare VDB 10

[Idclip]

Main changes:

• New VFX minium of Y2021
• Minium ABI support is now 8. Support for ABI 7 has been removed
• Support for Houdini 18.5 is removed. Support for Houdini 19.5 is added.
• NanoVDB minimum tested CUDA version is now 11 (to support GCC >= 7.0)
• Minimum GCC version is now 9.3
• Minimum support C++ standard is now 17
• Minimum supported CMake version is now 3.18
• Minimum supported Python version is not 3.7

Some temporary changes which need to be resolved:

• Temporarily had to remove NanoVDB from the address sanitizer tests due to some compiler failures
• Minimum supported Blosc version is now 1.17. Blosc tests have temporarily been disabled until VDB containers are bumped https://github.com/AcademySoftwareFoundation/aswf-docker/issues/156
• Houdini builds on VFX 2021 docker images are failing: https://github.com/Idclip/openvdb/runs/7723189745?check_suite_focus=true

[Idclip]

Looks great, thanks Nick! C++17, woo!

Happy for this to go in when we can get the CI to pass.

I don't know why the Houdini builds are failing - @jmlait @e4lam have you seen this error before:

https://github.com/AcademySoftwareFoundation/openvdb/runs/7723876851?check_suite_focus=true

I suspect it may be to do with how boost is being built but this only occurs when building against Houdini 19/19.5 on the VFX2021 image:

usr/local/bin/clang++ -O3 -DNDEBUG CMakeFiles/vdb_print.dir/main.cc.o -o vdb_print   -L/github/home/houdini_install/hou/dsolib  -Wl,-rpath,/github/home/houdini_install/hou/dsolib:/__w/openvdb/openvdb/build/openvdb/openvdb: ../../openvdb/openvdb/libopenvdb.so.10.0.0 /github/home/houdini_install/hou/dsolib/libjemalloc.so /github/home/houdini_install/hou/dsolib/libtbb.so.2 /github/home/houdini_install/hou/dsolib/libz.so -pthread /usr/local/lib/libboost_iostreams.so.1.73.0 
/opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/../../../../bin/ld: /usr/local/lib/libboost_iostreams.so.1.73.0: undefined reference to `lzma_cputhreads@XZ_5.2.2'
/opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/../../../../bin/ld: /usr/local/lib/libboost_iostreams.so.1.73.0: undefined reference to `lzma_stream_encoder_mt@XZ_5.1.2alpha'

[e4lam]

@Idclip We've built Houdini's boost iostreams against xz 5.2.3 for liblzma for quite a while now even back in H18.5. This has not changed in H19 nor H19.5.

$ ldd libhboost_iostreams-mt-x64.so.1 | grep lzma
        liblzma.so.5 => .../Nightly18.5CMake/dev/hfs/dsolib/./liblzma.so.5 (0x00007f50b874e000)

Looking at your error messages though, it looks like there are references to two different XZ versions (and neither of them matches Houdini's 5.2.3) so I'm not sure what's happening. I'd hazard a guess that something changed in the VFX2021 image. It might be worthwhile comparing the /usr/local/lib/libboost_iostreams.so.1.73.0 dependencies in the VFX2021 image with the ones in the older image.

PS. You can double-check the XZ versions in $HFS/custom/include/lzma/version.h

[Idclip]

Thanks @e4lam, It's just odd that we have builds on the VFX2021 image using the same-ish settings but without linking against Houdini libs which are passing:

fail: https://github.com/AcademySoftwareFoundation/openvdb/runs/7723876851?check_suite_focus=true pass: https://github.com/AcademySoftwareFoundation/openvdb/runs/7723872597?check_suite_focus=true

failed link:

/usr/local/bin/clang++ -O3 -DNDEBUG CMakeFiles/vdb_print.dir/main.cc.o -o vdb_print -L/github/home/houdini_install/hou/dsolib -Wl,-rpath,/github/home/houdini_install/hou/dsolib:/__w/openvdb/openvdb/build/openvdb/openvdb: ../../openvdb/openvdb/libopenvdb.so.10.0.0 /github/home/houdini_install/hou/dsolib/libjemalloc.so /github/home/houdini_install/hou/dsolib/libtbb.so.2 /github/home/houdini_install/hou/dsolib/libz.so -pthread /usr/local/lib/libboost_iostreams.so.1.73.0

passed link:

/usr/local/bin/clang++ -O3 -DNDEBUG CMakeFiles/vdb_print.dir/main.cc.o -o vdb_print -Wl,-rpath,/__w/openvdb/openvdb/build/openvdb/openvdb: ../../openvdb/openvdb/libopenvdb.so.10.0.0 /usr/local/lib/libtbbmalloc.so.2 /usr/local/lib/libtbb.so.2 /usr/lib64/libz.so -pthread /usr/local/lib/libboost_iostreams.so.1.73.0

The other main difference is the allocator in use. Will have to dig into this a bit more.

[e4lam]

@Idclip I don't think the allocator will matter. Again, I'd double-check your /usr/local/libboost_iostreams.so.1.73.0. It might be that it has been incorrectly built all along but we just never noticed it until something actually uses boost iostreams. This is because the linker might have previously discarded unused functions that would try to link against lzma.

Another possibility might be that different lzma headers are being included leading to erroneous references to those lzma functions. However, the linker is complaining about it directly in boost iostreams, which suggests this issue already happened when building boost iostreams itself. Run nm on it to see if the references are in fact what the linker says. For example in H19.0, I see this:

$ nm libhboost_iostreams.so | grep lzma_cputhreads
                 U lzma_cputhreads@@XZ_5.2

[e4lam]

@Idclip Ah, our resident Linux expert pointed out something blindingly obvious that I had missed. Your linker line links against boost iostreams, but since boost iostreams depends on lzma, then you should also (explicitly) have liblzma.so on the linker line itself. But again the fact that the linker is trying to match against two different XZ versions might suggest that there is also an issue with how VFX2021's boost was built.

[Idclip]

@Idclip I don't think the allocator will matter

It was more if it's actually a weird linker script doing something funky.

Your linker line links against boost iostreams, but since boost iostreams depends on lzma, then you should also (explicitly) have liblzma.so on the linker line itself.

Of course, but again, this only happens with the houdini builds. All other builds, using the exact same boost library, are linking and running correctly. We're not lazyly binding anything and clang should be erroring with undefined symbols in binaries anyway, which means that the passed link commands are 100% correct. Perhaps some change of order of symbol resolution is causing the issue but we're using the boost cmake modules which isn't pulling in an explicit lzma link - implying that the downstream boost cmake stuff also thinks this is fine. This is also true of zlib, zstd and bz2:

https://www.boost.org/doc/libs/1_79_0/libs/iostreams/doc/installation.html

edit: I think you're on to something with the Houdini headers causing this. Here's the build command:

cd /__w/openvdb/openvdb/build/openvdb_cmd/vdb_print && /usr/local/bin/ccache /usr/local/bin/clang++ -DBOOST_ALL_NO_LIB -DBOOST_IOSTREAMS_DYN_LINK -DOPENVDB_DLL -DOPENVDB_USE_DEPRECATED_ABI_8 -I/__w/openvdb/openvdb/openvdb/openvdb/.. -I/__w/openvdb/openvdb/build/openvdb/openvdb -I/__w/openvdb/openvdb/build/openvdb/openvdb/openvdb -isystem /github/home/houdini_install/hou/toolkit/include -O3 -DNDEBUG -pthread -std=c++17 -o CMakeFiles/vdb_print.dir/main.cc.o -c /__w/openvdb/openvdb/openvdb_cmd/vdb_print/main.cc

If they are taking precedent over a lzma installed in /usr/include then maybe the system boost is doing something weird with the symbols it's trying to use.

[Idclip]

The more I think about it the more my comment doesn't make sense either... just need to debug it

[Idclip]

We had a chat about this in the TSC. We think it's to do with some cmake injecting the houdini dsolib as a search folder with -L/github/home/houdini_install/hou/dsolib and causing ld to select liblzma from there rather than the system version. From what I can see there are only two places this is coming from, both in our CMake which is setting INTERFACE_LINK_DIRECTORIES on targets:

https://github.com/AcademySoftwareFoundation/openvdb/blob/33813dbef87734a7f0f70ca4cfed69474bce84fe/cmake/FindTBB.cmake#L436 https://github.com/AcademySoftwareFoundation/openvdb/blob/33813dbef87734a7f0f70ca4cfed69474bce84fe/cmake/FindBlosc.cmake#L377

I'm not sure if either of these are necessary anymore but will first verify this resolves the problem.

[Idclip]

Looks great, thanks Nick! C++17, woo!

Happy for this to go in when we can get the CI to pass.

I decided to just disable the H19.5 builds for now but the H19 builds won't build with C++17 on the VFX2020 platform (and it's the VFX2021 platform which is causing the issue). So I can get this passing but we'd have to temporarily disable all the Houdini CI whilst we figure out what's going on.