De-vendor CMake dependencies and use shared libraries

[gigony]

This patch de-vendors CMake dependencies and use/link shared libraries as much as possible.

• https://github.com/rapidsai/cucim/issues/686

• Switch to build with CXX11 ABI because we are now building manylinux_2_28 wheel.

  • Previously, we set _GLIBCXX_USE_CXX11_ABI=0 because manylinux2014 requires gcc4-compatible libstdcxx-abi(gcc is configured with '--with-default-libstdcxx-abi=gcc4-compatible', https://gcc.gnu.org/onlinedocs/libstdc++/manual/configure.html) which forces to set _GLIBCXX_USE_CXX11_ABI=0 so std::pmr::string wouldn't be available on CentOS 7.

• Update tests/unit/clara/test_image_cache.py::test_get_shared_memory_cache to reduce the shared memory size.

  • https://github.com/rapidsai/cucim/actions/runs/10203511389/job/28230563808?pr=753
    • FAILED tests/unit/clara/test_image_cache.py::test_get_shared_memory_cache - RuntimeError: No space left on device

Tasks

• [ ] Update CMake files to use the conda package (and fall back to the downloaded source if the CONDA_PREFIX environment variable is not available).
  • [ ] cpp/cmake/deps
    • [x] abseil.cmake
    • [x] boost-header-only.cmake
    • [x] catch2.cmake => Using Catch2 from Conda package has some issues with the CMake integration.
    • [x] cli11.cmake
    • [x] fmt.cmake
    • [ ] googlebenchmark.cmake
    • [ ] googletest.cmake
    • [x] libcuckoo.cmake => no conda package available
    • [x] nlohmann_json.cmake
    • [x] nvtx3.cmake
    • [x] taskflow.cmake
  • [ ] python/cmake/deps
    • [x] fmt.cmake
    • [x] nlohmann_json.cmake
    • [x] pybind11_json.cmake
    • [x] pybind11.cmake
  • [ ] cpp/plugins/cucim.kit.cuslide/cmake/deps
    • [x] catch2.cmake
    • [x] cli11.cmake
    • [x] fmt.cmake
    • [ ] googlebenchmark.cmake
    • [ ] googletest.cmake
    • [ ] libculibos.cmake
    • [ ] libdeflate.cmake
    • [ ] libjpeg-turbo.cmake
    • [ ] libopenjpeg.cmake
    • [ ] libtiff.cmake
    • [x] nlohmann_json.cmake
    • [ ] nvjpeg.cmake
    • [ ] openslide.cmake
    • [ ] pugixml.cmake
  • [ ] cpp/plugins/cucim.kit.cumed/cmake/deps
    • [x] catch2.cmake
    • [x] cli11.cmake
    • [x] fmt.cmake
    • [ ] googlebenchmark.cmake
    • [ ] googletest.cmake
• [ ] Make sure that required shared libraries are embedded in the xx.libs folder in the Python wheel file:
  • Related to this file
  • Currently, for https://files.pythonhosted.org/packages/c1/be/e25edc72728620c51fbab884a473a37e440323ad9f555750a267a4801e08/cucim_cu12-24.6.0-cp311-cp311-manylinux_2_28_x86_64.whl, the following files are embedded:
    • cucim_cu12.libs/liblzma-51a76f52.so.5.2.4
    • cucim_cu12.libs/libcudart-b5a066d7.so.12.2.140
    • cucim_cu12.libs/libcucim-61bee0fe.so.24.06.00
• [ ] Switch to use rapids-cmake

Added conda packages (for build)

• boost-cpp
• libabseil
• fmt
• nlohmann_json
• nvtx-c // https://github.com/rapidsai/cucim/issues/168
• taskflow // https://taskflow.github.io/taskflow/Releases.html
• pybind11
• pybind11_json
• catch2
  • Add C++17 support that "just works" via package managers · Issue #2462 · catchorg/Catch2
  • https://stackoverflow.com/a/70320798/16361228
• cli11

Test procedure

using docker

# On the repository root folder
docker run \
  --rm \
  -it \
  --gpus all \
  --pull=always \
  --network=host \
  --volume $PWD:/repo \
  --workdir /repo \
  rapidsai/ci-conda:cuda12.5.1-ubuntu22.04-py3.10


# Inside the docker

## remove build cache
find . -type d -name 'build-release' -exec rm -rf {} +
find /opt/conda/conda-bld/work/ -type d -name 'build-release' -exec rm -rf {} +

## run ci locally
./ci/build_cpp.sh

using conda locally

mamba env create -n cucim -f conda/environments/all_cuda-125_arch-x86_64.yaml
mamba activate cucim

./run build_local clean && ./run build_local clean_local && ./run build_local all release $CONDA_PREFIX

[copy-pr-bot[bot]]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[gigony]

/ok to test

[jakirkham]

Thanks Gigon! 🙏

Moved this to 24.10. Hope that is ok 🙂

[jakirkham]

/ok to test

[gigony]

/ok to test

[gigony]

/ok to test

[jakirkham]

• googlebenchmark.cmake

In conda-forge this is called benchmark

Though I wonder if we want to use rapids-cmake's rapids_cpm_gbench

[gigony]

/ok to test

[gigony]

/ok to test

[jakirkham]

libculibos.cmake

cuLIBOS is a static library that is included as part of CUDART and installed with the NVCC compiler

In terms of finding it, CMake already has FindCUDAToolkit, which includes a CUDA::culibos target that we can require and add to dependencies

So we can probably drop this file and simplify associated logic

[gigony]

/ok to test

[gigony]

@jakirkham It looks like the build container is out of space.

https://github.com/rapidsai/cucim/actions/runs/10206351504/job/28239540987?pr=753

  FAILED tests/unit/clara/test_image_cache.py::test_get_shared_memory_cache - RuntimeError: No space left on device

[jakirkham]

Do we know how much space we are using?

[jakirkham]

/ok to test

[jakirkham]

Style check failure is due to a small typo. Fixing in PR: https://github.com/rapidsai/cucim/pull/759

[jakirkham]

Rebased

Edit: Hoped that would mean GitHub would sign all the commits for us, but it seems that didn't work

[jakirkham]

/ok to test

[jakirkham]

We discussed this PR in the cuCIM meeting and decided this wasn't going to make 24.10. So have moved to 24.12

[gigony]

https://github.com/rapidsai/cucim/pull/785#issuecomment-2387189833 for devendoring libtiff

[jakirkham]

Retargeted to 25.04

Note there are some conflicts introduced from PR: https://github.com/rapidsai/cucim/pull/834