MOSS 使用fnlp/moss-moon-003-sft-int4和fnlp/moss-moon-003-sft-int8模型报错

因为GPU显存32G，所以修改了文件moss_gui_demo.py 中的模型引用，改为fnlp/moss-moon-003-sft-int4和fnlp/moss-moon-003-sft-int8后都报错。

~/MOSS$ python moss_gui_demo.py Downloading pytorch_model.bin: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 10.8G/10.8G [04:29<00:00, 40.0MB/s] Fetching 14 files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 [04:29<00:00, 19.27s/it] Waiting for all devices to be ready, it may take a few minutes... Traceback (most recent call last): File "moss_gui_demo.py", line 34, in model = load_checkpoint_and_dispatch( File "/home/good/anaconda3/envs/moss/lib/python3.8/site-packages/accelerate/big_modeling.py", line 479, in load_checkpoint_and_dispatch load_checkpoint_in_model( File "/home/good/anaconda3/envs/moss/lib/python3.8/site-packages/accelerate/utils/modeling.py", line 893, in load_checkpoint_in_model raise ValueError(f"{checkpoint} is not a folder containing a .index.json file.") ValueError: /home/good/.cache/huggingface/hub/models--fnlp--moss-moon-003-sft-int4/snapshots/b60bb82d458373c51bfb48106991caf924c281ff is not a folder containing a .index.json file.

Apr 25 '23 07:04 ImGoodBai

其实是少了.index.json这个文件，没办法索引到模型。看非量化模型的文件夹是有多这个文件的。改地址不行...头疼

Apr 25 '23 07:04 SitaraJin

少了文件这个提示的很清楚，但我要如何才能正确切换使用int8的模型呢？

Apr 25 '23 07:04 ImGoodBai

痛苦，我的报错也是这样

Apr 25 '23 07:04 nanyu0564

需要更换model赋值的语句,替换为：model = MossForCausalLM.from_pretrained("/root/moss-moon-003-sft-int8", trust_remote_code=True).half().cuda()。可以参考这个：https://blog.csdn.net/genghaojie123/article/details/130357804