lm-evaluation-harness icon indicating copy to clipboard operation
lm-evaluation-harness copied to clipboard

Published 20 hours ago • verified publisher icon EleutherAI

Fix slow gguf tests

Open For-rest2005 opened this issue 7 months ago • 1 comments
trafficstars

In this pull request, we add a new API model, "builtin_gguf". It is derived from https://github.com/abetlen/llama-cpp-python/issues/1983 But we can't get a correct results directly, due to some errors on the API of llama-cpp-python. I have post an issue on llama-cpp-python project. In this issue, I give my solution for the specific need in "builtin_gguf" and make some optimization for these APIs. You can modify the code in llama-cpp-python manually according to the issue. But my work can only meet our needs when running within "builtin_gguf". It is not thorough for llama-cpp-python. Thus I do not make a pull request. It is wait for others to solve. By the way, this API-model is still not complete. Hope you guys can complement it.

For-rest2005 avatar Mar 26 '25 15:03 For-rest2005

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Mar 26 '25 15:03 CLAassistant