运行webui，如果打开--load_8bit True 就会出来这个错误

[thewintersun]

--load_8bit True 会出来这个错误， 如果False就不会有错误，但是结果是乱码。

Traceback (most recent call last): File "/home/jovyan/dluo/github/LaWGPT/webui.py", line 211, in fire.Fire(main) File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "/home/jovyan/dluo/github/LaWGPT/webui.py", line 42, in main model = LlamaForCausalLM.from_pretrained( File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2784, in from_pretrained ) = cls._load_pretrained_model( File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3125, in _load_pretrained_model new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model( File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/transformers/modeling_utils.py", line 717, in _load_state_dict_into_meta_model set_module_8bit_tensor_to_device( File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/transformers/utils/bitsandbytes.py", line 78, in set_module_8bit_tensor_to_device new_value = bnb.nn.Int8Params(new_value, requires_grad=False, has_fp16_weights=has_fp16_weights).to(device) File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/bitsandbytes/nn/modules.py", line 227, in to return self.cuda(device) File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/bitsandbytes/nn/modules.py", line 191, in cuda CB, CBt, SCB, SCBt, coo_tensorB = bnb.functional.double_quant(B) File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/bitsandbytes/functional.py", line 1642, in double_quant row_stats, col_stats, nnz_row_ptr = get_colrow_absmax( File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/bitsandbytes/functional.py", line 1531, in get_colrow_absmax lib.cget_col_row_stats(ptrA, ptrRowStats, ptrColStats, ptrNnzrows, ct.c_float(threshold), rows, cols) File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/ctypes/init.py", line 387, in getattr func = self.getitem(name) File "/home/jovyan/.conda/envs/lawgpt/lib/python3.10/ctypes/init.py", line 392, in getitem func = self._FuncPtr((name_or_ordinal, self)) AttributeError: /home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cget_col_row_stats

如果设置是False就出来的答案莫名其妙：

GPU环境：

[herobrine19]

您好, 可能是bitsandbytes库的问题.您可以试一下这样 cd ~/.conda/envs/lawgpt/lib/python3.10/site-packages/bitsandbytes cp libbitsandbytes_cuda110.so libbitsandbytes_cpu.so

[thewintersun]

似乎没有起作用，出来的错误如下：

python webui.py --base_model ./models/base_models/chinese-llama-7b-merged/ --lora_weights ./models/lora_weights/legal-lora-7b/ --prompt_template "law_template" --load_8bit True

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

bin /home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so /home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: /home/jovyan/.conda/envs/lawgpt did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths... warn(msg) CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... /home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/lib64')} warn(msg) /home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)! warn(msg) CUDA SETUP: Highest compute capability among GPUs detected: 6.1 CUDA SETUP: Detected CUDA version 117 /home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU! warn(msg) CUDA SETUP: Loading binary /home/jovyan/.conda/envs/lawgpt/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so... Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]Error named symbol not found at line 528 in file /mmfs1/gscratch/zlab/timdettmers/git/bitsandbytes/csrc/ops.cu

[herobrine19]

您好, 试试这样 cd ~/.conda/envs/lawgpt/lib/python3.10/site-packages/bitsandbytes cp libbitsandbytes_cuda117_nocublaslt.so libbitsandbytes_cpu.so

[thewintersun]

可以了