gpt-tokenizer icon indicating copy to clipboard operation
gpt-tokenizer copied to clipboard

Published 20 hours ago • verified publisher icon niieani

chore: setup o200k_base tokenizer

Open mbukeRepo opened this issue 1 year ago • 0 comments
trafficstars

This PR integrates the new tokenizer for gpt-4o o200k_base.

Encoding file: https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken 214

Closes #42

mbukeRepo avatar May 15 '24 12:05 mbukeRepo