Role tags for local models

[giorgiopiatti]

According to the documentation role tags are also supported, as shown in the chatbot example here: https://lmql.ai/#chat.

By looking around in the code I see that this is only the case for ChatGPT tags, I think it will be very helpful to included a model information object which can be passed when loading the model and where users can specify the corresponding translation.

Since each model uses different encoding scheme, we will need something similar to what is done on FastChat's LLM Arena (see fastchat/conversation.py.

I'm available to do the coding for this and submit a pull request!

[lbeurerkellner]

Hi there, thanks a lot for offering help. We definitely would welcome role tag support with other models.

As a pointer for coding, you can have a look at this function:

https://github.com/eth-sri/lmql/blob/main/src/lmql/runtime/interpreter.py#L465

All prompt components are pre-processed by this function, and we already replace lmql:ROLE/ tags for non-OpenAI Chat models. To support other models, you can start by implementing a string-to-string transformation there, transforming lmql:ROLE/ annotated prompts to the respective e.g. [INST]...[/INST] format. Let me know if you need more info or pointers :)

[stevenbedrick]

So I just ran into this issue and so spent a few minutes digging around in the code to figure out what was going on... another thought besides doing string-to-string would be to leverage the apply_chat_template() function provided by HF's tokenizer, but this might not be as robust as one would like. It does seem to need to be something on a model-specific basis, as unfortunately there seems to be a lot of variation between models in terms of how they want things formatted. Mistral's models, for example, need something kind of funky in that it seems to interact weirdly with their sentence start and stop tokens.