HanishKVC comments

Results 37 comments of


                                            HanishKVC

Implement (properly) different chat templates in main.cpp

I think we should see if we can have a generic chat-apply-template logic, independent of the different handshake-template standards, which inturn is driven by a json file, which provides the...

Generic Chat templating code with text/json file based config; main chat updated to drive its in-prefix, in-suffix and reverse-prompt from same; chat-apply-template equivalent c-api to allow use by other codes also

> This is interesting. The only issue I see with this is that it doesn't account for FIM (Fill-in-the-Middle). Other than that, it seems alright. > > Something to note...

Generic Chat templating code with text/json file based config; main chat updated to drive its in-prefix, in-suffix and reverse-prompt from same; chat-apply-template equivalent c-api to allow use by other codes also

> > This is interesting. The only issue I see with this is that it doesn't account for FIM (Fill-in-the-Middle). Other than that, it seems alright. > > Something to...

Generic Chat templating code with text/json file based config; main chat updated to drive its in-prefix, in-suffix and reverse-prompt from same; chat-apply-template equivalent c-api to allow use by other codes also

Updated notes ## Overview Helps chat with a model, by allowing role based special token tagging, based on the specified chat-handshake-template-standard. This is used by main, to build on existing...

Generic Chat templating code with text/json file based config; main chat updated to drive its in-prefix, in-suffix and reverse-prompt from same; chat-apply-template equivalent c-api to allow use by other codes also

Sample chaton_meta.json includes template info for * llama2 * llama3 * gemma * chatml * zephyr * deepseek I noticed some difference between deepseek's actual tokenizer config and what is...

Generic Chat templating code with text/json file based config; main chat updated to drive its in-prefix, in-suffix and reverse-prompt from same; chat-apply-template equivalent c-api to allow use by other codes also

> > Or are you meaning coding related models and I dont know, if they have some fill-in-the-blank or is it fill-in-the-middle > > Yes, [this is what I meant](https://arxiv.org/abs/2207.14255)....

Generic Chat templating code with text/json file based config; main chat updated to drive its in-prefix, in-suffix and reverse-prompt from same; chat-apply-template equivalent c-api to allow use by other codes also

Have added support for Begin and Prefix entries wrt User role and inturn one can configure both of them individually wrt whether either of them get added to the 1st...

Generic Chat templating code with text/json file based config; main chat updated to drive its in-prefix, in-suffix and reverse-prompt from same; chat-apply-template equivalent c-api to allow use by other codes also

> At first lance, I'm not sure if it's a good idea to move the implementation completely into a separated JSON. While the good point is that it allows users...

Generic Chat templating code with text/json file based config; main chat updated to drive its in-prefix, in-suffix and reverse-prompt from same; chat-apply-template equivalent c-api to allow use by other codes also

Also as json library seems to be co-opted into llama.cpp/common, so I used the same and built my concept on top of it actually. If the logic in this PR...

Generic Chat templating code with text/json file based config; main chat updated to drive its in-prefix, in-suffix and reverse-prompt from same; chat-apply-template equivalent c-api to allow use by other codes also

@ngxson hope below gives some more background and or info on the idea behind this PR > I've just have a look in detail for this PR. The idea seems...