WIP: Add metatensor

[jhossbach]

Closes #4933

[RudolfWeeber]

Progress:

• Sketch of the main code for model loading and running with Lammps dependenceis removed. This will not compile. Just skteches. Also, sno Espresso specific code, yet.
• adding dependnceis in CMake. I failed to ge fetchcontent working. In the end, I built both metatnesor-core and metatensor-torch externally.

[RudolfWeeber]

Notes on building:

Torhc

wget "https://download.pytorch.org/libtorch/cu121/libtorch-cxx11-abi-shared-with-deps-2.4.0%2Bcu121.zip"

unzip libtorch-cxx11-abi-shared-with-deps-2.4.0+cu121.zip 

Note that this is libtorch, not torch propper. Everything is pre-compiled. Dependencies included. the archive is huge. There is a cpu-only version of this as well, which, hopefully, is small enough to be included in CI.

Intsall rust

I used the rustup script from the rust installatoin page, which installs to ~..cargo.

Source rust config

. ~/.cargo/env

metatensor

git clone from lab-cosmo/metatensor cd metatensor/metatensor-coe cmake . -DCMAKE_INSTALL_PREFIX=~/es_mt_deps # or other user-owned target dir make make install

## metatensor-torch

cd metatensor/metatensor-torch cmake . -DTorch_DIR=whereever/libtorch/share/cmake/Torch -dmetatensor_DIR=~/es_mt_deps/lib/cmake/metatensor make make install

## Espresso


Some of the following is ICP specific. Main points are torhc and metatensor dirs.

```bash
GCC_VER=12; CUDA_VER=12.0; CC=gcc-${GCC_VER} CXX=g++-${GCC_VER} CUDACXX=/usr/local/cuda-${CUDA_VER}/bin/nvcc CUDAARCHS="61;75;86" cmake ~/es -D ESPRESSO_BUILD_WITH_CUDA=ON -D CUDAToolkit_ROOT=/usr/local/cuda-${CUDA_VER} -D ESPRESSO_BUILD_WITH_CCACHE=ON -D ESPRESSO_BUILD_WITH_WALBERLA=ON -D ESPRESSO_BUILD_WITH_WALBERLA_FFT=ON -D ESPRESSO_BUILD_WITH_WALBERLA_AVX=ON ESPRESSO_BUILD_WITH_HDF5=OFF -D ESPRESSO_BUILD_WITH_GSL=ON -D CMAKE_CUDA_FLAGS="--compiler-bindir=/usr/bin/g++-${GCC_VER}" \
   -DESPRESSO_BUILD_WITH_METATENSOR=on \
   -DTorch_DIR=/ssd/weeber/libtorch/share/cmake/Torch\
   -Dmetatensor_DIR=~/es_mt_deps/lib/cmake/metatensor\
   -Dmetatensor_torch_DIR=~/es_mt_deps/lib/cmake/metatensor_torch

[RudolfWeeber]

Forgot two points:

• A small patch to metatensor is needed, seehttps://github.com/lab-cosmo/metatensor/pull/712
• Espresso uses CUDA (from feature configuraiton) as a macro. This causes a compiler error in torch. Before including anything toch, #undef CUDA.

[jhossbach]

(Also do not forget to append libtorch/lib and espresso_metatensor_deps/lib to the LD_LIBRARY_PATH)

[ceriottm]

hello there, we're putting the finishing touches on a paper on the "metatensor ecosystem" and we would very much love to mention also the integration in ESPRESSO. is there any way we can help to get this merged?

[PicoCentauri]

I just had a very nice offline discussion with @jhossbach and @RudolfWeeber and we try to get this going again.

[jhossbach]

Current setup that builds (on my machine):

export PATH=/usr/local/cuda-12.4/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-12.4/lib64:$LD_LIBRARY_PATH
export CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-12.4

rm -rf build
mkdir build
cd build

source ../.venv/bin/activate

echo $CMAKE_PREFIX_PATH
TORCH_PREFIX=$(python -c "import torch; print(torch.utils.cmake_prefix_path)")
MTS_PREFIX=$(python -c "import metatensor; print(metatensor.utils.cmake_prefix_path)")
MTS_TORCH_PREFIX=$(python -c "import metatensor.torch; print(metatensor.torch.utils.cmake_prefix_path)")
MTA_TORCH_PREFIX=$(python -c "import metatomic.torch; print(metatomic.torch.utils.cmake_prefix_path)")
CMAKE_PREFIX_PATH="$TORCH_PREFIX;$MTS_PREFIX;$MTS_TORCH_PREFIX;$MTA_TORCH_PREFIX"
echo $CMAKE_PREFIX_PATH

cmake ../ \
  -D ESPRESSO_BUILD_WITH_CUDA=OFF \
  -D ESPRESSO_BUILD_WITH_CCACHE=OFF \
  -D ESPRESSO_BUILD_WITH_WALBERLA=OFF \
  -D ESPRESSO_BUILD_WITH_WALBERLA_FFT=OFF \
  -D ESPRESSO_BUILD_WITH_WALBERLA_AVX=OFF \
  -D ESPRESSO_BUILD_WITH_GSL=OFF \
  -D ESPRESSO_BUILD_WITH_METATENSOR=ON \
  -D CMAKE_PREFIX_PATH="$CMAKE_PREFIX_PATH"

[espresso-ci]

Your pull request does not meet our code documentation rules. The Doxygen documentation generated 2 unique warnings (total: 4, ignored: 2):

src/core/ml_metatensor/sketches_espresso_bindings/system.hpp:
  line 13: Found ';' while parsing initializer list! ()
  line 17: Found ';' while parsing initializer list! ()

You can generate these warnings with make -t; make doxygen; ../maintainer/CI/dox_warnings.sh using the maxset config. This is the same command that I have executed to generate the log above.

[espresso-ci]

Your pull request does not meet our code formatting rules. To fix this, please do one of the following:

• You can download a patch with my suggested changes here, inspect it and make changes manually.
• You can directly apply it to your repository by running curl https://gitlab.icp.uni-stuttgart.de/espressomd/espresso/-/jobs/401683/artifacts/raw/style.patch | git apply -.
• You can run maintainer/CI/fix_style.sh to automatically fix your coding style. This is the same command that I have executed to generate the patch above, but it requires certain tools to be installed on your computer.

You can run gitlab-runner exec docker style afterwards to check if your changes worked out properly.

Please note that there are often multiple ways to correctly format code. As I am just a robot, I sometimes fail to identify the most aesthetically pleasing way. So please look over my suggested changes and adapt them where the style does not make sense.