openai-tools icon indicating copy to clipboard operation
openai-tools copied to clipboard

Published 20 hours ago • verified publisher icon dluc

Does GPT4 use a different tokenizer?

Open joelmartinez opened this issue 1 year ago • 1 comments

Given that this API uses a very explicit GPT3Tokenizer name ... I was just curious if this would work for GPT4. Thanks!

joelmartinez avatar Aug 14 '23 13:08 joelmartinez

@joelmartinez It will work, but it will be just slightly high. See: https://platform.openai.com/tokenizer

Not sure this is still being updated.

kinetiq avatar Oct 24 '23 17:10 kinetiq