openai-tokens
openai-tokens copied to clipboard
fix: add missing encoder for 4o models
When using a gpt-4o
as a model, the current release is failing when trying to get an encoder (all the unit tests use 3.5-turbo
as a model)
TypeError: Cannot read properties of undefined (reading 'encode')
If we look at js-tiktoken, gpt-4o
uses o200k_Base
as an encoder which isn't listed in the model list of encoder.js
.
This Pull Request adds the new o200k_Base
in to the encoder list. I added a simple test to make sure a variety of models return an encoder. Ideally I would have liked to have looped the const models
from models.js
but that isn't exported and I wasn't confident in changing that to be exported.
I also bumped the dependency to .14
as this is the minimum js-tiktoken
release that supports 4o
models. I think that's better than just relying on the ^
?