transformers icon indicating copy to clipboard operation
transformers copied to clipboard

Published 20 hours ago • verified publisher icon huggingface

update based on tokenizers release

Open ArthurZucker opened this issue 9 months ago • 4 comments

What does this PR do?

A small improvement, but overall allows us to add tokens special and not special at the same time for fast tokenizers

ArthurZucker avatar Apr 30 '24 13:04 ArthurZucker