chat-with-gpt icon indicating copy to clipboard operation
chat-with-gpt copied to clipboard

Published 20 hours ago • verified publisher icon cogentapps

Support for ggerganov/llama.cpp

Open ricklamers opened this issue 1 year ago • 5 comments

Would be cool if it could support https://github.com/ggerganov/llama.cpp

Is it on your roadmap?

ricklamers avatar Mar 14 '23 13:03 ricklamers

Maybe using https://cocktailpeanut.github.io/dalai/#/

ricklamers avatar Mar 14 '23 13:03 ricklamers

Integrating with the Dalai API should be doable.

How well does Llama perform in a chat style / conversational format?

cogentapps avatar Mar 14 '23 14:03 cogentapps

with https://alpaca-ai-custom5.ngrok.io it seems to perform well

schmidp avatar Mar 15 '23 21:03 schmidp

Alpaca is explicitly trained for Instruct-style messages. LLAMA by default isn't ready for conversations (yet) without some clever prompt engineering.

rounakdatta avatar Mar 16 '23 04:03 rounakdatta

I'd like to implement this via Dalai's API which now supports Alpaca, but this issue needs resolved first:

https://github.com/cocktailpeanut/dalai/issues/51

It's also a bit slow for longer chats. Ideally it would use llama.cpp's interactive mode instead of resending the whole context with each message, but I don't think this is implemented yet?

cogentapps avatar Mar 22 '23 11:03 cogentapps