FAQ (please read before opening a new issue)

[hauntsaninja]

trafficstars

Usage help

Check out this awesome tokeniser app https://tiktokenizer.vercel.app/ built by Diagram!

Check out the OpenAI cookbook!

In particular, the following are great examples of using tiktoken:

• How to count tokens with tiktoken
• How to format inputs to ChatGPT models
• Embedding long inputs
• Entity extraction for long documents

Other useful resources:

• OpenAI documentation
• OpenAI community forum
• OpenAI discord
• Contact [email protected]

Support for other languages

• See https://github.com/openai/tiktoken/issues/97 for suggestions

Publishing an official NPM package

I have no current plans to do this, but it is likely that in the future OpenAI will provide an official package. Discuss at #22

Publishing an official Rust crate

In the future when I have time, I will publish the core BPE logic as its own crate. Discuss at #24

I'm short on time currently, but in the longer term I want to explore ways to make maintaining downstream bindings easier. Apologies to the people who've reached out about this — y'all are great, sorry for being a bottleneck.

Packaging

Issues with pyinstaller

I've never used pyinstaller, but people report --hidden-import=tiktoken_ext.openai_public --hidden-import=tiktoken_ext helps make things work. See #43 for more details.

Wheels for other platforms

I'm reluctant to publish wheels for additional platforms, particularly if I can't easily build or test code for them. I'm unlikely to do this unless there's significant interest. Please add reactions to the following issues if you'd like to see those wheels. No promises, but I'll reopen any issue that gets >10 reactions.

• https://github.com/openai/tiktoken/pull/65
• https://github.com/openai/tiktoken/issues/86
• https://github.com/openai/tiktoken/issues/94