rtp-llm icon indicating copy to clipboard operation
rtp-llm copied to clipboard
verified publisher icon alibaba

support ebedding和reranker 部署时量化8

Open invisifire opened this issue 1 year ago • 3 comments

invisifire avatar Nov 07 '24 10:11 invisifire

w8a8还是weight only，weight only暂时还没支持

ySingularity avatar Nov 18 '24 08:11 ySingularity

weight only

也就是现在我外部量化8后ebedding是支持量化部署的？

invisifire avatar Nov 18 '24 09:11 invisifire

等我们下次更新补充这部分文档说明 直接转的ckpt格式可能会有点问题

ySingularity avatar Nov 21 '24 07:11 ySingularity