TigerBot
TigerBot copied to clipboard
TigerResearch
ModuleNotFoundError: No module named 'exllamav2'
The test environment: the server OS is Ubuntu 22.04 and there is a T4 GPU available. (1)did below install and infer steps according to https://huggingface.co/TigerResearch/tigerbot-13b-chat-4bit-exl2: conda create --name tigerbot python=3.8 conda activate tigerbot conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia git clone https://github.com/TigerResearch/TigerBot cd TigerBot pip install -r requirements.txt
git clone https://github.com/turboderp/exllamav2 cd exllamav2 pip install -r requirements.txt cd ..
CUDA_VISIBLE_DEVICES=0 python other_infer/exllamav2_hf_infer.py --model_path TigerResearch/tigerbot-13b-chat-4bit-exl2
(2)Got the error:
Traceback (most recent call last):
File "other_infer/exllamav2_hf_infer.py", line 9, in
(3)Please advise how to fix this issue.
try "pip install exllamav2"? and "pip show exllamav2" to check if installed successfully
(1)Did:
pip install exllamav2
CUDA_VISIBLE_DEVICES=0 python other_infer/exllamav2_hf_infer.py --model_path TigerResearch/tigerbot-13b-chat-4bit-exl2
(2)Got:
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda-12.1'
Traceback (most recent call last):
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/exllamav2/ext.py", line 14, in
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "other_infer/exllamav2_hf_infer.py", line 9, in
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1308, in load
return _jit_compile(
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1710, in _jit_compile
_write_ninja_file_and_build_library(
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1810, in _write_ninja_file_and_build_library
_write_ninja_file_to_build_library(
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 2199, in _write_ninja_file_to_build_library
cuda_flags = common_cflags + COMMON_NVCC_FLAGS + _get_cuda_arch_flags()
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1980, in _get_cuda_arch_flags
arch_list[-1] += '+PTX'
IndexError: list index out of range
Hello, you can upgrade your Python version to 3.9 first and try again. We work normally with python=3.10 and CUDA version=11.6. The recommended Python version is >=3.9. We will also investigate the solution as soon as possible.
I think I found the reason. When switched to Python 3.8, I encountered the same problem as you. I realized that it might be due to a mismatch between the torch version and the CUDA version. Therefore, I reinstalled torch under Python 3.8 and ensured that it was consistent with the current CUDA version. The model loaded smoothly. Good Luck !
conda create --name tigerbot python=3.8
conda activate tigerbot
#pytorch official installation
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia
git clone https://github.com/TigerResearch/TigerBot
cd TigerBot
pip install -r requirements.txt
git clone https://github.com/turboderp/exllamav2
cd exllamav2
pip install -r requirements.txt
pip install exllamav2
cd ..
CUDA_VISIBLE_DEVICES=0 python other_infer/exllamav2_hf_infer.py --model_path TigerResearch/tigerbot-13b-chat-4bit-exl2
loading model: TigerResearch/tigerbot-13b-chat-4bit-exl2...
Traceback (most recent call last):
File "other_infer/exllamav2_hf_infer.py", line 300, in
pip install torch==2.0.1+cu117 torchvision torchaudio --extra-index-url https://mirror.sjtu.edu.cn/pytorch-wheels/cu117
yes, it's a bug. exllamav2 will first check local path. You can download model weights from HF manually. We will repair it soon.
Thank you! Another option is to do 4 bits quantization with AutoGPTQ only.