David Duong

Is there a necessity for a Windows script? As there isn't a pre-install.sh script anymore.

@darknoon Sure thing, created a separate PR for better code readability (AFAIK Github doesn't allow showing PR to different repositories). https://github.com/eisber/tiktoken/pull/2 I think it would be better to keep this...

Hello, I've been working on JS bindings for tiktoken, found here: https://github.com/dqbd/tiktoken. Core methods are implemented, some methods are missing for now. You can install it as such: ``` npm...

Hello @w-biggs! The issue stems from the invalid type, which will be fixed in #5403. In the meantime, you can either: 1. Ignore the error with `@ts-expect-error` directive ```ts {...

Hi @gablabelle! Thanks for the contribution. I've tried to use the new App Router as well, but it does seem like there's some upstream issue with React Server Components (RSC)...

Hello! the OpenAI tokenizer does not treat special tokens as a single tokens, whereas the Tiktokenizer does when selecting `gpt-3.5-turbo`. You can get the same behaviour by selecting `cl100k_base` or...

Hi @ted-at-openai! Sounds fine by me, glad that the tool is useful and helpful to this day 🙂 ---- In general the end-goal for this tool is to merge the...

However, the best solution would be to use `unstable_wasm` of `huggingface/tokenizers` to support SentencePiece, Unigram and other tokenizers.