tiktoken
tiktoken copied to clipboard
Tiktoken not published to cargo
It seems that the tiktoken package is not linkable from Rust using Cargo's default registry.
Are there plans to publish the tiktoken
crate? Is it published on another registry?
Thanks for your work on this BPE encoder, I've already found it very useful!
Repro:
In a rust project, run
cargo add tiktoken
Expected behavior:
Cargo should find, download and add tiktoken
to the available crates
Actual behavior:
$ cargo add tiktoken
Updating crates.io index
error: the crate `tiktoken` could not be found in registry index.