ctransformers icon indicating copy to clipboard operation
ctransformers copied to clipboard

Does ctransformers boost the inference speed in llm inference?

Open pradeepdev-1995 opened this issue 1 year ago • 0 comments

I have converted my finetuned hugging face model to .gguf format and triggered the inference with ctransformers. I am using a CUDA GPU machine. But i did not observe any kind of inference speed improvement after the inference by ctransformers. Observing the same latency in transformer based infernce and ctransformer based inference.

pradeepdev-1995 avatar Feb 15 '24 13:02 pradeepdev-1995