sentencepiece Tokenizer

[kyleroche]

Describe the solution you'd like tokenizer for https://github.com/google/sentencepiece/blob/master/python/README.md

This would be used for Google Vertex AI Prompt driver

[derekpai]

Evaluate in a month

[collindutter]

Closing as we ended up integrating with google generative AI which provides a tokenizer.