openai-tokens icon indicating copy to clipboard operation
openai-tokens copied to clipboard

fix: add missing encoder for 4o models

Open randompixel opened this issue 4 months ago • 1 comments

When using a gpt-4o as a model, the current release is failing when trying to get an encoder (all the unit tests use 3.5-turbo as a model)

TypeError: Cannot read properties of undefined (reading 'encode')

If we look at js-tiktoken, gpt-4o uses o200k_Base as an encoder which isn't listed in the model list of encoder.js.

This Pull Request adds the new o200k_Base in to the encoder list. I added a simple test to make sure a variety of models return an encoder. Ideally I would have liked to have looped the const models from models.js but that isn't exported and I wasn't confident in changing that to be exported.

I also bumped the dependency to .14 as this is the minimum js-tiktoken release that supports 4o models. I think that's better than just relying on the ^?

randompixel avatar Oct 01 '24 18:10 randompixel