transformers icon indicating copy to clipboard operation
transformers copied to clipboard

Add split special tokens

Open itazap opened this issue 1 year ago • 1 comments

What does this PR do?

Fixes #30685 #28648

Before submitting

  • [ ] make sure this is saved and used not only as kwargs but also the attribute
  • [ ] added tests
  • [ ] TODO: add docs

Edit:

  • [ ] Moved split_special_token testing to common tests, as it should be supported by both fast and slow models.
  • [ ] Verify fast and slow behaviour is identical
  • [ ] Tested with T5
  • [ ] Note: split_special_tokens is expected in kwargs

itazap avatar May 13 '24 09:05 itazap

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.