Open-Assistant icon indicating copy to clipboard operation
Open-Assistant copied to clipboard

Add tokenizer config classes

Open jackapbutler opened this issue 2 years ago • 2 comments

Fixes #990

I've just refactored out the tokenizer configuration details from the get_tokenizer function. This should hopefully;

  • Make it easier to add new tokenizers without changing logic in get_tokenizer.
  • Make it easier to see what get_tokenizer is doing without having to worry about the special tokens of each model type.

jackapbutler avatar Feb 06 '23 17:02 jackapbutler

:x: pre-commit failed. Please run pre-commit run --all-files locally and commit the changes. Find more information in the repository's CONTRIBUTING.md

github-actions[bot] avatar Feb 06 '23 17:02 github-actions[bot]

:x: pre-commit failed. Please run pre-commit run --all-files locally and commit the changes. Find more information in the repository's CONTRIBUTING.md

github-actions[bot] avatar Feb 06 '23 17:02 github-actions[bot]