gpt-tokenizer icon indicating copy to clipboard operation
gpt-tokenizer copied to clipboard

Published 20 hours ago • verified publisher icon niieani

Is the algorithm of this repo consistent with openai tiktoken?

Open Zwe1 opened this issue 7 months ago • 0 comments

Hi, Is the algorithm of this repo consistent with openai tiktoken? It's important for us to keep consistency with openai

Zwe1 avatar Nov 12 '23 12:11 Zwe1