请问text-generation-webui报Cannot copy out of meta tensor; no data

[purewater2011]

感谢您使用Issue提问模板，请按照以下步骤提供相关信息。我们将优先处理信息相对完整的Issue，感谢您的配合。

提示：将[ ]中填入x，表示打对钩。提问时删除上面这两行。请只保留符合的选项，删掉其他。

问前必查项目

• [x] 由于相关依赖频繁更新，请确保按照Wiki中的相关步骤执行
• [x] 我已阅读FAQ章节并且已在Issue中对问题进行了搜索，没有找到相似问题和解决方案
• [x] 第三方插件问题：例如llama.cpp、text-generation-webui、LlamaChat等，同时建议到对应的项目中查找解决方案

选择问题类型

基础模型：

• [x] LLaMA
• [x] Alpaca

问题类型：

• [x] 模型量化和部署问题（llama.cpp、text-generation-webui、LlamaChat）

详细描述问题

python server.py --model llama-13b-hf --lora chinese-alpaca-lora-13b --gpu-memory 20 --share --auto-devices 运行后进行推理时提示异常： Traceback (most recent call last): File "/project/text-generation-webui/modules/callbacks.py", line 66, in gentask ret = self.mfunc(callback=_callback, **self.kwargs) File "/project/text-generation-webui/modules/text_generation.py", line 252, in generate_with_callback shared.model.generate(**kwargs) File "/home/eric/anaconda3/envs/lora-13b/lib/python3.9/site-packages/peft/peft_model.py", line 716, in generate outputs = self.base_model.generate(**kwargs) File "/home/eric/anaconda3/envs/lora-13b/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/home/eric/anaconda3/envs/lora-13b/lib/python3.9/site-packages/transformers/generation/utils.py", line 1485, in generate return self.sample( File "/home/eric/anaconda3/envs/lora-13b/lib/python3.9/site-packages/transformers/generation/utils.py", line 2524, in sample outputs = self( File "/home/eric/anaconda3/envs/lora-13b/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/eric/anaconda3/envs/lora-13b/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) File "/home/eric/anaconda3/envs/lora-13b/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 700, in forward logits = self.lm_head(hidden_states) File "/home/eric/anaconda3/envs/lora-13b/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/eric/anaconda3/envs/lora-13b/lib/python3.9/site-packages/accelerate/hooks.py", line 160, in new_forward args, kwargs = module._hf_hook.pre_forward(module, *args, **kwargs) File "/home/eric/anaconda3/envs/lora-13b/lib/python3.9/site-packages/accelerate/hooks.py", line 280, in pre_forward set_module_tensor_to_device(module, name, self.execution_device, value=self.weights_map[name]) File "/home/eric/anaconda3/envs/lora-13b/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 149, in set_module_tensor_to_device new_value = value.to(device) NotImplementedError: Cannot copy out of meta tensor; no data!

运行截图或log

[wtwong316]

I hit the same issue on python3.10.

[wtwong316]

For zzcgithub, do you mention the step 4.

修改/modules/LoRA.py文件，大约在第28行

shared.model.resize_token_embeddings(len(shared.tokenizer)) shared.model = PeftModel.from_pretrained(shared.model, Path(f"{shared.args.lora_dir}/{lora_name}"), **params)

No change for the second line, otherwise it fails. I only add the first line.

[wtwong316]

I hit the same issue on python3.10.

I discover that if I run with --cpu, then there is no issue. However, the performance (the response) is really slow, slower than tortoise. I also check the error "Cannot copy out of meta tensor; no data" on a gpu case. Someone just mention not enough memory for the GPU. I have one A30 card with 24GB memory.

Anyone know how many GPU memory needed for the 13b to work with? I can run 7b with no issue but it is terrible for the response.

[iMountTai]

First, you can directly load the merged chinese-alpaca model using the webui, which is faster than loading two weight files. Second, the weight of 13B model itself is 24G, and A30 cannot fit properly. For more details and parameters of webui, please pay attention to the text-generation-webui. Thank you for your attention. In addition, if the 7B model does not perform well, has prompt inference been added?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your consideration.

[greatliu]

Hi, Author, I have met the same problem...

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your consideration.

[github-actions[bot]]

Closing the issue, since no updates observed. Feel free to re-open if you need any further assistance.