gpt-tokenizer
gpt-tokenizer copied to clipboard
niieani
JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features.
When we use gpt-tokenizer pkg packageging fails. The warning in build: ``` Warning Cannot find module 'gpt-tokenizer/model/gpt-3.5-turbo' from 'D:\****\explorer' in D:\***\explorer.js > Warning Cannot include file %1 into executable. The...
Hi, Is the algorithm of this repo consistent with openai tiktoken? It's important for us to keep consistency with openai
I am experiencing 200MB increase after implementing `gpt-tokenizer` the only function that I am using from this library is `isWithinTokenLimit`. Here is an image of my memory consumtion before and...
How to set the model in `encode` / `encodeGenerator` like `encodeChat` or `encodeChatGenerator`? ```ts import { encode } from 'gpt-tokenizer' const text = 'Hello, world!' const generator =encodeGenerator(text, /* I...
Hello there, I'm trying to use `gpt-tokenizer` inside a NextJs edge function and get the following error ```ts Attempted import error: './index.js' does not contain a default export (imported as...
Unsure if this is just a documentation issue, however, after checking the source code there appears to be no default or clear instruction on how to provide the model type...
hello, i use it in rn project, it took a mistake ``` ReferenceError: Property 'TextDecoder' doesn't exist, js engine: hermes at SharePage (http://localhost:8081/src\pages\share\index.bundle?platform=android&app=com.awesomeproject&modulesOnly=true&dev=true&minify=false&runModule=true&shallow=true:12:26) at RNGestureHandlerRootView at GestureHandlerRootView at App at...
把包引入项目后，运行项目报错。 错误信息:Error: Invariant: Method expects to have requestAsyncStorage, none available Error: Invariant: Method expects to have requestAsyncStorage, none available at Module.headers (webpack-internal:///(sc_server)/./node_modules/next/dist/esm/client/components/headers.js:23:15) at handleReqBailout (webpack-internal:///(sc_server)/./node_modules/next/dist/server/future/route-modules/app-route/helpers/proxy-request.js:76:35) at Object.get (webpack-internal:///(sc_server)/./node_modules/next/dist/server/future/route-modules/app-route/helpers/proxy-request.js:96:13) at requestOpenai...
We found that this module is not ready for CJK characters， when type `ここに内容を入力すると、消費されるメダルの数が計算されます。` OpenAI show: This module show The token is different to OpenAI.
From the documentation seems possible to import functions directly from gpt-tokenizer as: ``` import { encodeChat, isWithinTokenLimit } from "gpt-tokenizer/esm/main" ``` But doing it and running that code will create...
