autolabel icon indicating copy to clipboard operation
autolabel copied to clipboard

Published 20 hours ago • verified publisher icon refuel-ai

.get_num_tokens() for RefuelLLM

Open DhruvaBansal00 opened this issue 2 years ago • 1 comments

DhruvaBansal00 avatar Nov 27 '23 17:11 DhruvaBansal00

Currently, .get_num_tokens() just returns the length of the string for RefuelLLM. Let's change this to use the tokenizer and return the correct number of tokens.

DhruvaBansal00 avatar Nov 27 '23 17:11 DhruvaBansal00