Linly icon indicating copy to clipboard operation
Linly copied to clipboard

Published 20 hours ago • verified publisher icon CVI-SZU

能不能提供一份量化模型int4的方法或者脚本？

Open doctor1984 opened this issue 1 year ago • 5 comments

doctor1984 avatar Apr 28 '23 05:04 doctor1984

先转成llama的格式，再用llama.cpp的脚本做量化，可以量化成int4 int5 int8

riverzhou avatar May 08 '23 08:05 riverzhou

先转成llama的格式，再用llama.cpp的脚本做量化，可以量化成int4 int5 int8

请问bin文件转成pth的方法有脚本吗？麻烦分享一个呗，十分感谢

doctor1984 avatar May 09 '23 09:05 doctor1984

可以参考： https://github.com/Tencent/TencentPretrain/blob/main/scripts/convert_tencentpretrain_to_llama.py

fengyh3 avatar May 09 '23 09:05 fengyh3

可以参考： https://github.com/Tencent/TencentPretrain/blob/main/scripts/convert_tencentpretrain_to_llama.py

非常感谢，我试试哈，谢谢。

doctor1984 avatar May 10 '23 00:05 doctor1984

可以参考： https://github.com/Tencent/TencentPretrain/blob/main/scripts/convert_tencentpretrain_to_llama.py

P01son/Linly-ChatFlow-13B他这个模型转换int4后依然不能使用

doctor1984 avatar May 10 '23 00:05 doctor1984