gpt-tokenizer
gpt-tokenizer copied to clipboard
niieani
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o. Port of OpenAI's tiktoken with additional features.
As listed in the [`README.md`](https://github.com/niieani/gpt-tokenizer#supported-models-and-their-encodings), there is no support for the new models: * `gpt-3.5-turbo-0613` * `gpt-3.5-turbo-16k-0613`
This PR integrates the new tokenizer for gpt-4o `o200k_base`. Encoding file: [https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken 214](https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken) Closes #42
https://github.com/openai/tiktoken/commit/c0ba74c238d18b4824c25f3c27fc8698055b9a76
[](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [rfc4648](https://togithub.com/swansontec/rfc4648.js) | [`^1.5.2` -> `^1.5.3`](https://renovatebot.com/diffs/npm/rfc4648/1.5.2/1.5.3) | [](https://docs.renovatebot.com/merge-confidence/)...
![renovate[bot] avatar](https://avatars.githubusercontent.com/in/2740?v=4 "renovate[bot] avatar"){kind=avatar}
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github/codeql-action](https://redirect.github.com/github/codeql-action) | action | major | `v2` -> `v3` | --- ### Release...
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [actions/setup-node](https://redirect.github.com/actions/setup-node) | action | major | `v3` -> `v4` | --- ### Release...
This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [@niieani/scaffold](https://redirect.github.com/niieani/scaffold) | [`^1.7.32` -> `^1.7.34`](https://renovatebot.com/diffs/npm/@niieani%2fscaffold/1.7.32/1.7.34) | [](https://docs.renovatebot.com/merge-confidence/) | [](https://docs.renovatebot.com/merge-confidence/)...
This PR contains the following updates: | Update | Change | |---|---| | lockFileMaintenance | All locks refreshed | 🔧 This Pull Request updates lock files to use the latest...
