llama-cpp-python
llama-cpp-python copied to clipboard
abetlen
Count/truncate number of tokens before processing
Create a function that takes in text as input, converts it into tokens, counts the tokens, and then returns the text with a maximum length that is limited by the token count. The returned text will be truncated if it exceeds the specified token count, ensuring that it does not exceed the maximum context size.
@jakvb what's the use case for this?
Something like:
def truncate(llama: Llama, input: str, maxlen: int) -> str:
return llama.detokenize(llama.tokenize(input)[:maxlen])
@jakvb If you're using langchain as a wrapper around your LlamaCpp model, you can count the number of tokens before calling the llm with the following method get_num_tokens
def get_num_tokens(self, text: str) -> int:
tokenized_text = self.client.tokenize(text.encode("utf-8"))
return len(tokenized_text)
Example usage
num_tokens: int = llama_model.get_num_tokens(question)
And if we don't use LangChain?
# Where llm is an instance of the Llama class with a model loaded
tokens = llm.tokenize(b"Q: Name the planets in the solar system? A: ")
print(len(tokens))
How would you truncate the final dictionary that includes the system etc?
I actually haven't tested if this includes the system prompt. One quick and easy way to test this and resolve it if it does would be to tokenize an empty string. If you get tokens back it includes the system prompt, if it doesn't then it doesn't.
Then you can just subtract the length of the system prompt dictionary from the start of the final dictionary.
My untested guess would be that using this function directly would not include the system prompt, as you pass that along with your call to the completion function. I would assume the completion function calls this internally with the system prompt prepended. I can't test any of this at the moment but it should be pretty easy to figure out.