machinelearning icon indicating copy to clipboard operation
machinelearning copied to clipboard

Published 20 hours ago • verified publisher icon dotnet

Add GPT 4.5 to Tiktoken Tokenizer

Open luisquintanilla opened this issue 9 months ago • 0 comments

OpenAI introduced GPT 4.5.

It's not clear yet what the tokenizer used for it is but when that's known, updates need to be made to TikToken Tokenizer.

System Card

Related to issue: openai/tiktoken#385

luisquintanilla avatar Feb 27 '25 20:02 luisquintanilla