alpaca.cpp
alpaca.cpp copied to clipboard
antimatter15
Modifying chat.cpp to implement a ChatGPT-like conversational context memory
Is there any trivial way in which the code in chat.cpp could be tweaked to prepend each submitted prompt with the most recent couple of answers and prompts perhaps (or will this make the input sequence too large for the model?) so that alpaca could have some contextual understanding about the previous interactions within this conversation and reply accordingly as in ChatGPT. I would have liked to try it myself but I don't have enough expertise in C++ to attempt this. Any ideas or references?
If needed, I posted in this related issue a prompt template that handles the conversational context. And about C++, I can't help unfortunately.
I cant tell. Are you saying something needs to be written in the chat.exe app to do the context? I didn't know if you were brainstorming or this was something already implemented?
I'm confused by the original request. Are you saying it needs prior context? If so, I'm shocked, because I've been carrying on contiguous conversations in 30B with no issue wrt conversation context. Is it chatgpt level? Nope. But it's not missing.
I'm confused by the original request. Are you saying it needs prior context? If so, I'm shocked, because I've been carrying on contiguous conversations in 30B with no issue wrt conversation context. Is it chatgpt level? Nope. But it's not missing.
Really? My first few chats with the 7B model revealed a lack of conversational context awareness. Maybe I should just experiment more with it.
You can't really compare the performance conversationally of 7b to 30b. 7b just isn't very good. It can answer some questions but doesn't have the capability to hold context well. 30b, with the same chat.cpp is considerably better, but still not great. The issues you're having are probably more about the model than the cpp code in the project. That's what I'm trying to say. Also, from personal experience, 13b is actually worse than 7b... just a fair warning.
@Terristen Which CLI args do you use, please?
I wish I could tell you right now, but I'm mid build on a new bigger/badder machine so I don't have access to my latest .bat file. I definitely run -t 20 and turn the temperature up to 1.35 if I recall. As for others, I've not done a lot of tuning but I think I dial up the repeat last to something higher than default. My ongoing issue is running out of memory after about 25 prompts, though up to that it's pretty cogent.
I'm using LLAMA 60B and remember our conversation perfectly. LLAMA 7B is too simple to remember conversation. 60B is really good ....
My parameters are not fancy ...
main -m models/65B/ggml-model-q4_0.bin -t 16 -n 2048 --keep 48 --repeat_penalty 1.0 --color -i -r "User:" -f prompts/chat-with-bob.txt
alpaca.cpp is outdated anyway ... they merged to llama.ccp.
