openai-tokens
openai-tokens copied to clipboard
fix: add missing encoder for 4o models
When using a gpt-4o as a model, the current release is failing when trying to get an encoder (all the unit tests use 3.5-turbo as a model)
TypeError: Cannot read properties of undefined (reading 'encode')
If we look at js-tiktoken, gpt-4o uses o200k_Base as an encoder which isn't listed in the model list of encoder.js.
This Pull Request adds the new o200k_Base in to the encoder list. I added a simple test to make sure a variety of models return an encoder. Ideally I would have liked to have looped the const models from models.js but that isn't exported and I wasn't confident in changing that to be exported.
I also bumped the dependency to .14 as this is the minimum js-tiktoken release that supports 4o models. I think that's better than just relying on the ^?
Rebased branch based on latest chore(deps) updates on main
@mrsteele; would you accept this PR in to a bugfix release?
Looks great! Thanks for the fix!
:tada: This PR is included in version 2.3.6 :tada:
The release is available on:
Your semantic-release bot :package::rocket: