tiktoken
tiktoken copied to clipboard
openai
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
:wave: I'm an AI agent that writes, runs, and maintains Unit Tests. I even highlight the bugs I spot! I'm free for open-source repos. I started working from [make decoder...
Dear Developers, I'm pleased to inform you that I have completed the documentation update the load, model and registry files. The updated documentation provides clear explanations of function parameters, return...
The license name should be MIT. Instead, the name of the license is the whole license. This is a problem for our license checker.
Hello, **Token Usage Cost Calculation:** I would like to request the addition of a feature that calculates the cost of token usage based on OpenAI's current pricing. This feature would...
uses unsafe + std::mem::transmute to re-use encoder keys as decoder values and in the sorted_token_bytes list this should be safe because all these objects share a lifetime inside CoreBPE results...
use [rustc-hash](https://github.com/rust-lang/rustc-hash) for the HashSet too skips overhead of HashDoS mitigations