tiktoken
tiktoken copied to clipboard
openai
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
how to convert qwen.tiktoken to tokenzier.model
Not having a `Cargo.lock` file makes tiktoken harder to use by projects being packaged for deployment with [Nix](https://nixos.org/) (which runs builds in a sandbox that only has network access when...
## Description I'm trying to setup tiktoken locally with the help of this guide - https://stackoverflow.com/questions/76106366/how-to-use-tiktoken-in-offline-mode-computer. But the problem I'm facing is that the **.tiktoken** file - `cl100k_base.tiktoken` is getting...
See: * https://github.com/knuddelsgmbh/jtokkit/blob/main/lib/src/test/resources/cl100k_base_encodings.csv * https://github.com/knuddelsgmbh/jtokkit/blob/main/lib/src/test/resources/p50k_base_encodings.csv * https://github.com/knuddelsgmbh/jtokkit/blob/main/lib/src/test/resources/r50k_base_encodings.csv To make it simpler and smarter to input the parameters, instead of the CSV inputs I've pushed them as code. I've split out...
Fix #61 CC: @hauntsaninja
Continuing the optimizations started in https://github.com/openai/tiktoken/pull/237 and https://github.com/openai/tiktoken/pull/234, migrated from https://github.com/knuddelsgmbh/jtokkit/pull/75, https://github.com/knuddelsgmbh/jtokkit/pull/76, https://github.com/knuddelsgmbh/jtokkit/pull/77. This commit is mainly meant to address the issue of really big tokens spiraling out of control,...
I am using a freshly installed version of tiktoken==0.6.0 I am using the latest version of Anaconda with python 3.12.2 and trying to: ```import tiktoken``` and getting: ```ImportError: cannot import...
Is tiktoken supports **meta/llama2-70b**? I want to find the token count of the prompt before passing the prompt to a **meta/llama2-70b** model So how to do this with tiktoken ```...
This PR swaps `requests` for Python's built-in `urllib.request` library, reducing the library size a decent amount. Would you consider a PR doing something like this? Happy to make alterations.
Enterprises with internal networking configurations may want to limit access to outside resources for enterprise applications. Currently, `tiktoken` requires network access to `openaipublic.blob.core.windows.net` to populate the plugin modules which is...
