Sigbjørn Skjæret
Sigbjørn Skjæret
@compilade Any reason for not merging yet? If nix/flake changes are a concern they can be left out for now, ref https://github.com/ggml-org/llama.cpp/pull/13005#issuecomment-2831250849
Looks good, however it should not be in build.yml, it should be its own job that only runs when files in `vendor` are touched.
Sounds like you're just not using the correct chat template, try the one included in `models/templates/mistralai-Mistral-Nemo-Instruct-2407.jinja`.
I'm not sure what you mean by `mistral .jinja template`, the template I mentioned has tools.
This is where tools are added provided you pass `tools`, this is how all templates work: https://github.com/ggml-org/llama.cpp/blob/92bb442ad999a0d52df0af2730cd861012e8ac5c/models/templates/mistralai-Mistral-Nemo-Instruct-2407.jinja#L27-L50
So, if I understand correctly what you want is to enable tool handling even when `tools` is not provided?
> Why is mistral code block expliclty AFTER the test for tools, rather than some of the other cases? I'm pretty sure this was done on purpose at some point,...
Since the merging of a similar patch to `transformers` and the formalizing of `tools` (see references in #1336) there are starting to pop up a lot more models using `tojson`...
@abetlen [Llama 3.1](https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF) is the latest model to use `tojson`, would be nice to have matching behaviour as `transformers`.
@abetlen Even more `transformers` changes incoming, see updated OP.