ctransformers icon indicating copy to clipboard operation
ctransformers copied to clipboard
verified publisher icon marella

feature request

Open thistleknot opened this issue 2 years ago • 2 comments

batch inference

thistleknot avatar Sep 20 '23 02:09 thistleknot

I've been chasing gguf batch inference down, and apparently not supported in ctransformers, llama.cpp, nor llama-cpp-python

thistleknot avatar Sep 20 '23 02:09 thistleknot

Why?

yukiarimo avatar Dec 07 '23 08:12 yukiarimo