tiktoken-go-binding
tiktoken-go-binding copied to clipboard
j178
tiktoken-go
A Go binding for tiktoken-rs, for tokenizing text with OpenAI models using tiktoken.
Here are some pure Go implementations you might consider using instead of this one:
- https://github.com/pkoukk/tiktoken-go
- https://github.com/tiktoken-go/tokenizer
j178