transformers
transformers copied to clipboard
Add split special tokens
What does this PR do?
Fixes #30685 #28648
Before submitting
- [ ] make sure this is saved and used not only as kwargs but also the attribute
- [ ] added tests
- [ ] TODO: add docs
Edit:
- [ ] Moved split_special_token testing to common tests, as it should be supported by both fast and slow models.
- [ ] Verify fast and slow behaviour is identical
- [ ] Tested with T5
- [ ] Note: split_special_tokens is expected in kwargs
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.