autolabel
autolabel copied to clipboard
.get_num_tokens() for RefuelLLM
Currently, .get_num_tokens() just returns the length of the string for RefuelLLM. Let's change this to use the tokenizer and return the correct number of tokens.