Available version for 4-bit quantization supporting CUDA 11.1?

[justin4ai]

Hello, my server has a CUDA version of 11.1 but I am struggling to use the latest bitsandbytes package since it support CUDA from 11.7 ~ 12.5, where mine is outside the range.

Thus, I tried to manually downgrade the version step by step so that I could find the runnable version with my CUDA 11.1 near bitsandbytes==0.40.0. However, the following error occurs:

RuntimeError: Failed to import transformers.integrations.bitsandbytes because of the following error (look up to see its traceback):

        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

, where I found the solution was just to upgrade the package version.

After upgrading via pip install -U bitsandbytes, however,

AttributeError: /home/XXX/miniconda3/envs/XXX/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cquantize_blockwise_fp16_fp4

error occurs. The thing is, of course there is no .so file for cuda111 afterwards.

Anyone knows the proper bitsandbytes version for 4-bit quantization supporting CUDA 11.1?

Best, Junyeong Ahn