tiktoken-go

tiktoken-go copied to clipboard

Is there any plan to adapt the llama model?

3

maplessssy

I found this helpful to implement caching the output of the encoder within my own code. Apply if useful, otherwise feel free to ignore 😄

bakks

[Efficiency] Inefficient to create encoders multiple times

5

I noticed our tests were much slower when switching from https://github.com/tiktoken-go/tokenizer to this library. It seems to be because `tiktoken.EncodingForModel` is very slow (0.3s on my machine). Our I see...

omar-scio

FunkyYang

The title states the question.

arkadyb