OpenSearch benchmarks are failing

[julian-risch]

Our Haystack benchmarks take longer than usual ( 2d 20h instead of 1d 12h) and fail since Oct 8 because the opensearch benchmarks time out. Last successful run was on Oct 1. https://github.com/deepset-ai/haystack/actions/workflows/benchmarks.yml

Thank you @bogdankostic for pinging us about this.

[bogdankostic]

I suspect that this is related to the upgrade of PyTorch from version 2.0.1 (successful benchmark run) to 2.1.0 (failing benchmark run).

In the failing benchmark runs, we see the following warning message:

UserWarning: CUDA initialization: The NVIDIA driver on your system is too old (found version 11040). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org/ to install a PyTorch version that has been compiled with your version of the CUDA driver. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)

CML that is used for the benchmarks (see here for example) comes with CUDA 11.2, but it seems that PyTorch 2.1.0 requires CUDA 12.1.

[masci]

Closing as benchmarks were decommissioned