qlora
qlora copied to clipboard
artidoro
undefined symbol: cquantize_blockwise_fp16_fp4
AttributeError Traceback (most recent call last) Cell In[6], line 1 ----> 1 model = LlamaForCausalLM.from_pretrained("../hf_llama", device_map="auto", torch_dtype=torch.float16, load_in_4bit=True )
File ~/anaconda3/envs/qlora/lib/python3.11/site-packages/transformers/modeling_utils.py:2829, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs) 2819 if dtype_orig is not None: 2820 torch.set_default_dtype(dtype_orig) 2822 ( 2823 model, 2824 missing_keys, 2825 unexpected_keys, 2826 mismatched_keys, 2827 offload_index, 2828 error_msgs, -> 2829 ) = cls._load_pretrained_model( 2830 model, 2831 state_dict, 2832 loaded_state_dict_keys, # XXX: rename? 2833 resolved_archive_file, 2834 pretrained_model_name_or_path, 2835 ignore_mismatched_sizes=ignore_mismatched_sizes, 2836 sharded_metadata=sharded_metadata, 2837 _fast_init=_fast_init, 2838 low_cpu_mem_usage=low_cpu_mem_usage, ... --> 394 func = self._FuncPtr((name_or_ordinal, self)) 395 if not isinstance(name_or_ordinal, int): 396 func.name = name_or_ordinal
AttributeError: /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cquantize_blockwise_fp16_fp4
nd submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
bin /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so
/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
warn("The installed version of bitsandbytes was compiled without GPU support. "
/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /home/server/anaconda3/envs/qlora did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/lib64')}
warn(msg)
/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
warn(msg)
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 117
CUDA SETUP: Loading binary /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so...
loading base model EleutherAI/pythia-12b...
Downloading (…)l-00001-of-00003.bin: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9.81G/9.81G [03:57<00:00, 41.3MB/s]
Downloading (…)l-00002-of-00003.bin: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9.93G/9.93G [03:44<00:00, 44.3MB/s]
Downloading (…)l-00003-of-00003.bin: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.11G/4.11G [01:32<00:00, 44.5MB/s]
Downloading shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [09:20<00:00, 186.80s/it]
Loading checkpoint shards: 0%| | 0/3 [00:09<?, ?it/s]
Traceback (most recent call last):
File "/home/server/vocal/huggingface/qlora/qlora.py", line 758, in
@DamonGuzman Looks like you did not build bitsandbytes with gpu support? It is loading the cpu and not cuda version.
UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
CUDA SETUP: Loading binary /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so...
@DamonGuzman Looks like you did not build bitsandbytes with gpu support? It is loading the cpu and not cuda version.
UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
CUDA SETUP: Loading binary /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so...
That was actually really close to the problem I was having! I had created a new conda environment and needed to install cuda toolkit in that new environment. I always assumed cuda tool kit was a system-wide package but that doesn't seem to be the case.
Just in case this is helpful for someone:
If you get this with docker make sure to use an image with cuda toolkit installed, e.g.:
pytorch/pytorch:2.0.1-cuda11.7-cudnn8-devel
nd submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
bin /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable. warn("The installed version of bitsandbytes was compiled without GPU support. " /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32 /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /home/server/anaconda3/envs/qlora did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths... warn(msg) CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/lib64')} warn(msg) /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)! warn(msg) CUDA SETUP: Highest compute capability among GPUs detected: 8.6 CUDA SETUP: Detected CUDA version 117 CUDA SETUP: Loading binary /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so... loading base model EleutherAI/pythia-12b... Downloading (…)l-00001-of-00003.bin: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9.81G/9.81G [03:57<00:00, 41.3MB/s] Downloading (…)l-00002-of-00003.bin: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9.93G/9.93G [03:44<00:00, 44.3MB/s] Downloading (…)l-00003-of-00003.bin: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.11G/4.11G [01:32<00:00, 44.5MB/s] Downloading shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [09:20<00:00, 186.80s/it] Loading checkpoint shards: 0%| | 0/3 [00:09<?, ?it/s] Traceback (most recent call last): File "/home/server/vocal/huggingface/qlora/qlora.py", line 758, in train() File "/home/server/vocal/huggingface/qlora/qlora.py", line 590, in train model = get_accelerate_model(args, checkpoint_dir) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/vocal/huggingface/qlora/qlora.py", line 263, in get_accelerate_model model = AutoModelForCausalLM.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 472, in from_pretrained return model_class.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2829, in from_pretrained ) = cls._load_pretrained_model( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3172, in _load_pretrained_model new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/transformers/modeling_utils.py", line 718, in _load_state_dict_into_meta_model set_module_quantized_tensor_to_device( File "/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/transformers/utils/bitsandbytes.py", line 88, in set_module_quantized_tensor_to_device new_value = bnb.nn.Params4bit(new_value, requires_grad=False, **kwargs).to(device) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/nn/modules.py", line 176, in to return self.cuda(device) ^^^^^^^^^^^^^^^^^ File "/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/nn/modules.py", line 154, in cuda w_4bit, quant_state = bnb.functional.quantize_4bit(w, blocksize=self.blocksize, compress_statistics=self.compress_statistics, quant_type=self.quant_type) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/functional.py", line 776, in quantize_4bit lib.cquantize_blockwise_fp16_nf4(get_ptr(None), get_ptr(A), get_ptr(absmax), get_ptr(out), ct.c_int32(blocksize), ct.c_int(n)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/anaconda3/envs/qlora/lib/python3.11/ctypes/init.py", line 389, in getattr func = self.getitem(name) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/anaconda3/envs/qlora/lib/python3.11/ctypes/init.py", line 394, in getitem func = self._FuncPtr((name_or_ordinal, self)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cquantize_blockwise_fp16_nf4
I solved this by "conda install cudatoolkit=11.7 -y".
For me, I just replace libbitsandbytes_cpu.so to be libbitsandbytes_cuda117.so, which 117 is the cuda vision that I am now using.
You may refer to this link: https://github.com/TimDettmers/bitsandbytes/issues/156#issuecomment-1474056975 They have undefined symbol error but with a different symbol, and it can be solved by this method.
I faced the same issue today, turns it I had a version conflict between Cuda and PyTorch. A fresh install of Cuda and PyTorch did the trick for me.
you can try this method:
cd [your bitsandbytes env path]
eg. : cd /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/
you can copy the path from your terminal
then:
cp libbitsandbytes_cuda117.so libbitsandbytes_cpu.so
ps : "CUDA version 117" is your cuda version, so here is " libbitsandbytes_cuda117.so"
cp libbitsandbytes_cuda117.so libbitsandbytes_cpu.so
This method worked for me in WSL Ubuntu on Windows11. I installed anacoda3 and cuda118 special version for WSL2
FineTuning Vicuna-13B with QloRA works well on RTX4090
nd submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
bin /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable. warn("The installed version of bitsandbytes was compiled without GPU support. " /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32 /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /home/server/anaconda3/envs/qlora did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths... warn(msg) CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/lib64')} warn(msg) /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)! warn(msg) CUDA SETUP: Highest compute capability among GPUs detected: 8.6 CUDA SETUP: Detected CUDA version 117 CUDA SETUP: Loading binary /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so... loading base model EleutherAI/pythia-12b... Downloading (…)l-00001-of-00003.bin: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9.81G/9.81G [03:57<00:00, 41.3MB/s] Downloading (…)l-00002-of-00003.bin: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9.93G/9.93G [03:44<00:00, 44.3MB/s] Downloading (…)l-00003-of-00003.bin: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.11G/4.11G [01:32<00:00, 44.5MB/s] Downloading shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [09:20<00:00, 186.80s/it] Loading checkpoint shards: 0%| | 0/3 [00:09<?, ?it/s] Traceback (most recent call last): File "/home/server/vocal/huggingface/qlora/qlora.py", line 758, in train() File "/home/server/vocal/huggingface/qlora/qlora.py", line 590, in train model = get_accelerate_model(args, checkpoint_dir) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/vocal/huggingface/qlora/qlora.py", line 263, in get_accelerate_model model = AutoModelForCausalLM.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 472, in from_pretrained return model_class.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2829, in from_pretrained ) = cls._load_pretrained_model( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3172, in _load_pretrained_model new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/transformers/modeling_utils.py", line 718, in _load_state_dict_into_meta_model set_module_quantized_tensor_to_device( File "/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/transformers/utils/bitsandbytes.py", line 88, in set_module_quantized_tensor_to_device new_value = bnb.nn.Params4bit(new_value, requires_grad=False, **kwargs).to(device) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/nn/modules.py", line 176, in to return self.cuda(device) ^^^^^^^^^^^^^^^^^ File "/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/nn/modules.py", line 154, in cuda w_4bit, quant_state = bnb.functional.quantize_4bit(w, blocksize=self.blocksize, compress_statistics=self.compress_statistics, quant_type=self.quant_type) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/functional.py", line 776, in quantize_4bit lib.cquantize_blockwise_fp16_nf4(get_ptr(None), get_ptr(A), get_ptr(absmax), get_ptr(out), ct.c_int32(blocksize), ct.c_int(n)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/anaconda3/envs/qlora/lib/python3.11/ctypes/init.py", line 389, in getattr func = self.getitem(name) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/server/anaconda3/envs/qlora/lib/python3.11/ctypes/init.py", line 394, in getitem func = self._FuncPtr((name_or_ordinal, self)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: /home/server/anaconda3/envs/qlora/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cquantize_blockwise_fp16_nf4
I solved this by "conda install cudatoolkit=11.7 -y".
thank you very much, i solved the same problems as well with this function
