tiktoken
tiktoken copied to clipboard
openai
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
When I was inputting long text into a large model, that is, when the len of the text was 1024*1024, a StackOverflow error occurred. ``` thread '' panicked at src/lib.rs:227:33:...
This PR introduces a caching mechanism in `_encode_ordinary_native()`, which stores the tokens for each "piece" of text. When a piece of text is repeated, its tokens are retrieved from the...
Just adding a direct link toward https://pypi.org/project/tiktoken in the README file. Sometimes, it can be interesting to find the package on PyPI to find some key info. It is just...
### Key Features - Token Visualization: Display token, and their positions in the input text. - Interactive Interface: Allows users to input text and see the tokenized output in real-time....
0.7.0 had aarch64 musllinux wheel can the same be made for 0.8.0? thanks!
Getting the Permission denied error in FastApi application with langchain package deployed in IIS using HttpPlatformModule, can anyone let me know how to resolve this? When the app is hosted...
Due to the way these heavier processes work it may crash the first time it tries to build, presumably because it scedules all of them on the same node. If...
Tested gpt-4o and gpt-4.1 in the playground, and they seemed to use the same tokenizer because of the token count.
This PR proposes adding support for free-threaded Python. This requires bumping PyO3 to a newer version as support was added in version 0.23.0. If I'm not missing anything, the library...
