cohere-python icon indicating copy to clipboard operation
cohere-python copied to clipboard

Return token_strings in Client.tokenize() when offline=True

Open yifanmai opened this issue 1 year ago • 2 comments

Fixes #493

yifanmai avatar May 07 '24 22:05 yifanmai

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar May 07 '24 22:05 CLAassistant

Thanks @yifanmai for the contribution!

We intentionally skipped this when we introduced offline local tokenization; since we were optimizing for time. However, let me check how much overhead this adds (for bigger inputs) and if it's negligible we can add it.

abdullahkady avatar May 08 '24 11:05 abdullahkady

Noted, thanks!

yifanmai avatar Jan 07 '25 17:01 yifanmai