SallyBot icon indicating copy to clipboard operation
SallyBot copied to clipboard

Published 20 hours ago • verified publisher icon DeSinc

Koboldcpp LLM interface supported?

Open Vladonai opened this issue 1 year ago • 3 comments

This is a very popular method of interacting with models. Frequent updates and good performance. Can it be used with your program?

Vladonai avatar May 14 '23 06:05 Vladonai

Not currently but I may look into it some time.

Sorry late reply because gmail sends github emails to spam

DeSinc avatar Jun 02 '23 00:06 DeSinc

Actually, it occurs to me that the extension API which this was built for is deliberately compatible with the Kobold API, and it even runs on the same port number. Have you tried running sallybot with a kobold LLM running to see what happens? It might just already work right off the bat. Specifically, I think the format was made to be identical and cross-compatible with TavernAI or something like that, just from what I can remember of it, which uses KoboldAI to do its thing.

DeSinc avatar Jun 03 '23 06:06 DeSinc

I tried using koboldcpp by sending the request on port 5001/5005 or whatever the default is, and it actually sends the request and generates an output, but there's a bit of an error when it comes time for koboldcpp to send back the output back to the SallyBot code. It might be easy to solve but I just haven't been bothered to give it a look.

DeSinc avatar Jul 05 '23 06:07 DeSinc