llama.cpp
llama.cpp copied to clipboard
ggml-org
Add example script for rendering jinja2 templates
Sometimes I need to inspect the models chat templates and I created a script awhile back to do this. This is a updated and modified version of the same script.
It's useful for debugging and comprehending how the model creator might have intended the chat template to be rendered. I like being able to visualize these things and this script helps me do that.
Example usage:
$ python gguf-py/scripts/gguf-template.py -r /mnt/valerie/models/mistralai/Mistral-7B-Instruct-v0.2/ggml-model-f16.gguf
<s>[INST] Hello! [/INST]Hello! How may I assist you today?</s>[INST] Can you tell me what pickled mayonnaise is? [/INST]Certainly! What would you like to know about it?</s>
This isn't a high priority, I just thought it might be useful.
I'm thinking it would be better to just extract bos/eos from metadata instead of allowing user to set them from command line. I've been working on some improvements to GGUFReader which would make it a lot easier to do things like that, and also enable the reduction of a lot of code duplication across the bundled scripts due to missing basic functionality.
If it's OK by you I'll wait until this is merged and then submit a PR with those improvements and remove --bos and --eos parameters at the same time.
@CISC No need to ask me for permission. If you think it's good, try it out. Would love to know about the results.
Just checking, in case you had a particular use for swapping out BOS/EOS. A --no-bos could be useful I guess.
This script can be really useful for generating prompts for main as well (esp. if --user and --assistant options were added to create a conversation instead of using dummy data) - giving me ideas - I like it! :)