transformers icon indicating copy to clipboard operation
transformers copied to clipboard

[`TokenizationUtils`] add support for `split_special_tokens`

Open ArthurZucker opened this issue 1 year ago • 1 comments

What does this PR do?

Adds support for split_special_tokens for fast models as well

  • [ ] deprecate split_special_tokens for encode_special_tokens for API consistency
  • [ ] make sure this is saved and used not only as kwargs but also the attribute
  • [ ] add some tests
  • [ ] add some docs

ArthurZucker avatar Jan 22 '24 16:01 ArthurZucker

Wip, not urgent! Can already be done but it's not save / loaded

ArthurZucker avatar Feb 23 '24 09:02 ArthurZucker