Daulet Zhanguzin

PTAL @peter-evans

I've created [Go bindings](https://github.com/daulet/tokenizers) for the library, currently surfacing very small API surface, mainly to satisfy my needs, but obviously additional contributions are welcome.

Hmm, I could add support for scoped services, but do you mind sharing what is your use case for scoped instances? The main challenge is to have a reliable way...

Cohere and a lot of other models use [HuggingFace's tokenizer](https://github.com/huggingface/tokenizers/tree/main/tokenizers), so a drop in fix is to use the library for tokenization (just feed corresponding tokenizer config, e.g. [this](https://huggingface.co/CohereForAI/c4ai-command-r-v01/blob/main/tokenizer.json)) and...

PTAL @gregjhogan

PTAL @geohot