tiktoken icon indicating copy to clipboard operation
tiktoken copied to clipboard

JS port and JS/WASM bindings for openai/tiktoken

Results 36 tiktoken issues
Sort by recently updated
recently updated
newest added

TikToken was working fine encoding and decoding tokens for me then without any changes (just demoing what was working before) it failed with this error: ``` Unhandled Runtime Error Error:...

https://github.com/openai/tiktoken/commit/9d01e5670ff50eb74cdb96406c7f3d9add0ae2f8

Hi, I'm trying to run tiktoken-js without using npm, just pure JS. However, I can't find a working example - the [readme](https://github.com/dqbd/tiktoken/blob/main/js/README.md) contains node instructions. Could anyone provide code that...

There are several open issues noting that in the worst case BPE merge algorithm in js-tiktoken takes quadratic time in the number of input characters for certain pathological inputs. This...

This is a pretty cool library! I'm doing something like this: ``` while (enc.encode(raw).length > length) { raw = raw.slice(0, -1000) } ``` to truncate big data, however this is...

I just noticed that this package is around 13MB when unpackaged and I reached my AWS Lambda package size limit. This is absolutely too big for serverless deployment. So my...

I'm using the sample code for Cloudflare Workers and got these errors in VS Code: ``` Cannot find module 'tiktoken/lite/init' or its corresponding type declarations. Cannot find module './node_modules/tiktoken/lite/tiktoken_bg.wasm' or...

Placeholder issue, OpenAI will release it soon

It would be great to have 1:1 parity with the python implementation. Adding this function would be a step in that direction. I already created my own implementation: ``` //...