tiktoken
tiktoken copied to clipboard
JS port and JS/WASM bindings for openai/tiktoken
TikToken was working fine encoding and decoding tokens for me then without any changes (just demoing what was working before) it failed with this error: ``` Unhandled Runtime Error Error:...
https://github.com/openai/tiktoken/commit/9d01e5670ff50eb74cdb96406c7f3d9add0ae2f8
Hi, I'm trying to run tiktoken-js without using npm, just pure JS. However, I can't find a working example - the [readme](https://github.com/dqbd/tiktoken/blob/main/js/README.md) contains node instructions. Could anyone provide code that...
There are several open issues noting that in the worst case BPE merge algorithm in js-tiktoken takes quadratic time in the number of input characters for certain pathological inputs. This...
please help
This is a pretty cool library! I'm doing something like this: ``` while (enc.encode(raw).length > length) { raw = raw.slice(0, -1000) } ``` to truncate big data, however this is...
I just noticed that this package is around 13MB when unpackaged and I reached my AWS Lambda package size limit. This is absolutely too big for serverless deployment. So my...
I'm using the sample code for Cloudflare Workers and got these errors in VS Code: ``` Cannot find module 'tiktoken/lite/init' or its corresponding type declarations. Cannot find module './node_modules/tiktoken/lite/tiktoken_bg.wasm' or...
Placeholder issue, OpenAI will release it soon
It would be great to have 1:1 parity with the python implementation. Adding this function would be a step in that direction. I already created my own implementation: ``` //...