mlx-swift-examples
mlx-swift-examples copied to clipboard
ml-explore
Use chat template
Since https://github.com/huggingface/swift-transformers/pull/104, it's now possible to use the chat template from tokenizer_config.json. ~~I've updated LLMEval to use the chat template, but I noticed that the output from the Mistral models ends with <|im_end|>, even when this string is included in extraEOSTokens. Perhaps @pcuenca has an idea why this might be happening?~~
Using the chat templates with Llama 3.2 will result in an error until none ~~and tojson are~~ is implemented in the Jinja package: https://github.com/maiqingqiang/Jinja/pull/4
Using the chat templates with Llama 3.2 will result in an error until
noneandtojsonare implemented in the Jinja package: maiqingqiang/Jinja#4
OK, should we hold off on taking this until that is merged (and we can pick it up here)?
OK, should we hold off on taking this until that is merged (and we can pick it up here)?
Yes, since LLMEval includes model configs for Llama 3.2. In the meantime, this branch can be used to test Llama 3.2 with the work in progress on Jinja.
~~We had a similar issue with an EOS token at the end of the output of Gemma 2: https://github.com/ml-explore/mlx-swift-examples/issues/89~~
~~This shouldn't be appearing in the output if it's included in extraEOSTokens. It could be an issue with the tokenizer, because I can see when it's generating in my app that <|im_end|> is being produced as multiple tokens, and it sometimes even continues generating beyond this string.~~
The problem with Mistral 7B is resolved here: https://github.com/huggingface/swift-transformers/pull/134
Llama 3.2 1B and 3B and Mistral 7B now work with chat templates, thanks to the changes in swift-transformers and Jinja.