mlx-swift-examples
mlx-swift-examples copied to clipboard
ml-explore
some models fail to prepare tokens: No chat template
Model loaded -> id("mlx-community/phi-2-hf-4bit-mlx")
Error: chatTemplate("No chat template was specified")
For models that have a chat template this is fine, but for those that do not:
- I don't see a way to probe for the presence of a chat template
- nor a way to provide a default
I think we need to do the config-mutate path and inject a template where needed.
Base models don't usually have a chat template. Is there something like https://github.com/ml-explore/mlx-examples/blob/main/llms/mlx_lm/generate.py#L213C1-L214C1 in Swift Transformers?
Base models don't usually have a chat template. Is there something like https://github.com/ml-explore/mlx-examples/blob/main/llms/mlx_lm/generate.py#L213C1-L214C1 in Swift Transformers?
No, nothing like that: https://github.com/huggingface/swift-transformers/blob/main/Sources/Tokenizers/Tokenizer.swift#L359
Perhaps there should be? @pcuenca @maiqingqiang
Base models don't usually have a chat template. Is there something like https://github.com/ml-explore/mlx-examples/blob/main/llms/mlx_lm/generate.py#L213C1-L214C1 in Swift Transformers?
No, nothing like that: https://github.com/huggingface/swift-transformers/blob/main/Sources/Tokenizers/Tokenizer.swift#L359
Perhaps there should be? @pcuenca @maiqingqiang
I agree. Similar to https://github.com/huggingface/transformers/blob/33868a057c02f0368ba63bd1edb746be38fe3d90/src/transformers/tokenization_utils_base.py#L1628-L1629
Alternatively, it could capture throw TokenizerError.chatTemplate and use tokenizer.encode instead.
Per above I added a workaround -- it will try to apply the template, but if that throws an error it will fall back to a built in template and tokenization.