gpt-tokenizer
gpt-tokenizer copied to clipboard
niieani
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o. Port of OpenAI's tiktoken with additional features.
This issue lists Renovate updates and detected dependencies. Read the [Dependency Dashboard](https://docs.renovatebot.com/key-concepts/dashboard/) docs to learn more. ## Rate-Limited These updates are currently rate-limited. Click on a checkbox below to force...
![renovate[bot] avatar](https://avatars.githubusercontent.com/in/2740?v=4 "renovate[bot] avatar"){kind=avatar}
[](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [@niieani/scaffold](https://togithub.com/niieani/scaffold) | [`^1.7.1` -> `^1.7.31`](https://renovatebot.com/diffs/npm/@niieani%2fscaffold/1.7.1/1.7.31) | [](https://docs.renovatebot.com/merge-confidence/)...
[](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [yarn](https://togithub.com/yarnpkg/berry) ([source](https://togithub.com/yarnpkg/berry/tree/HEAD/packages/yarnpkg-cli)) | [`3.5.0` -> `3.8.4`](https://renovatebot.com/diffs/npm/yarn/3.5.0/3.8.4) |...
I tried to follow sample usage, but the line `encodeChat(messages)` is returning this warning in VS Code: ``` [{ ... "message": "Argument of type '{ role: string; content: string; }[]'...
As you may know, function calls are now supported by OpenAI, and the function call tokens will be taken into account. With some googling, I found the calculation [here](https://community.openai.com/t/how-to-calculate-the-tokens-when-using-function-call/266573/5). It...
Hello, I'm using the following: ``` import { encode, isWithinTokenLimit } from 'gpt-tokenizer/model/text-davinci-003'; ``` which seems to slow down the initialization, enough that I can't deploy to cloudflare workers with...
Hello. The new -vision model lets you pass images (which end up as tokens). To properly calculate token use for my stuff, I'd need to be able to pass my...
[](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [actions/checkout](https://togithub.com/actions/checkout) | action | major | `v3` -> `v4` | ---...
[](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [yarn](https://togithub.com/yarnpkg/berry) ([source](https://togithub.com/yarnpkg/berry/tree/HEAD/packages/yarnpkg-cli)) | [`3.5.0` -> `3.5.1`](https://renovatebot.com/diffs/npm/yarn/3.5.0/3.5.1) |...
[](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [tsx](https://tsx.is) ([source](https://togithub.com/privatenumber/tsx)) | [`^3.12.7` -> `^3.12.10`](https://renovatebot.com/diffs/npm/tsx/3.12.7/3.12.10) |...