transformers
transformers copied to clipboard
[`TokenizationUtils`] add support for `split_special_tokens`
What does this PR do?
Adds support for split_special_tokens
for fast models as well
- [ ] deprecate
split_special_tokens
forencode_special_tokens
for API consistency - [ ] make sure this is saved and used not only as kwargs but also the attribute
- [ ] add some tests
- [ ] add some docs
Wip, not urgent! Can already be done but it's not save / loaded