openai-tokens icon indicating copy to clipboard operation
openai-tokens copied to clipboard

fix: add missing encoder for 4o models

Open randompixel opened this issue 1 year ago • 1 comments

When using a gpt-4o as a model, the current release is failing when trying to get an encoder (all the unit tests use 3.5-turbo as a model)

TypeError: Cannot read properties of undefined (reading 'encode')

If we look at js-tiktoken, gpt-4o uses o200k_Base as an encoder which isn't listed in the model list of encoder.js.

This Pull Request adds the new o200k_Base in to the encoder list. I added a simple test to make sure a variety of models return an encoder. Ideally I would have liked to have looped the const models from models.js but that isn't exported and I wasn't confident in changing that to be exported.

I also bumped the dependency to .14 as this is the minimum js-tiktoken release that supports 4o models. I think that's better than just relying on the ^?

randompixel avatar Oct 01 '24 18:10 randompixel

Rebased branch based on latest chore(deps) updates on main

@mrsteele; would you accept this PR in to a bugfix release?

randompixel avatar Oct 10 '24 08:10 randompixel

Looks great! Thanks for the fix!

mrsteele avatar Oct 13 '24 00:10 mrsteele

:tada: This PR is included in version 2.3.6 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

github-actions[bot] avatar Oct 13 '24 00:10 github-actions[bot]