AndriaK comments

Repositories
Issues
Comments

Results 1 comments of


                                            AndriaK

Proposal: Faster `Whitespace` PreTokenizer Implementation (10–30% Speedup)

Hi @ArthurZucker 👋 Thanks for the encouragement! I’ve just submitted the PR here: [https://github.com/huggingface/tokenizers/pull/1822](https://github.com/huggingface/tokenizers/pull/1822) **Title:** *Faster Whitespace PreTokenizer (Drop-in Replacement)* It includes: * Identical behavior to the original * 10–30%...