Open-Assistant
Open-Assistant copied to clipboard
Add tokenizer config classes
Fixes #990
I've just refactored out the tokenizer configuration details from the get_tokenizer
function. This should hopefully;
- Make it easier to add new tokenizers without changing logic in
get_tokenizer
. - Make it easier to see what
get_tokenizer
is doing without having to worry about the special tokens of each model type.
:x: pre-commit failed.
Please run pre-commit run --all-files
locally and commit the changes.
Find more information in the repository's CONTRIBUTING.md
:x: pre-commit failed.
Please run pre-commit run --all-files
locally and commit the changes.
Find more information in the repository's CONTRIBUTING.md