Binwei Yao

icon indicating a website link https://bigbinnie.github.io

icon location 2nd-year CS PhD student at UW-Madison | Ex. SE student at SJTU | NLPer

Results 1 issues of Binwei Yao

Why the tokenizer is slower than tiktoken?

8 comment

Hi, I tried to use the GPT2 tokenizer of HF and TikToken, but I found TikToken is faster than HF. Could you explain why this might happen?