cohere-python
cohere-python copied to clipboard
Return token_strings in Client.tokenize() when offline=True
Fixes #493
Thanks @yifanmai for the contribution!
We intentionally skipped this when we introduced offline local tokenization; since we were optimizing for time. However, let me check how much overhead this adds (for bigger inputs) and if it's negligible we can add it.
Noted, thanks!