Calculate the number of tokens

[Yakiw]

I would like to ask if there is a way to count the number of tokens consumed when making requests through the API (using ollama)

`import requests

API_URL = "xxxxx/api/v1/prediction/aaa5cb74-d79f-49d9-a7c7-c71a1303b72a"

def query(payload): response = requests.post(API_URL, json=payload) return response.json()

output = query({ "question": "Hey, how are you?", })`

[Yakiw]

Hello @Yakiw

Issue: response validation, token counting logic, model specification, missing tokenization, placeholder for the API URL, and insufficient error handling.

Enhancements: Include error handling, validate responses, include token counting using a tokenizer, and identify the appropriate tokenizer.

import requests
from transformers import AutoTokenizer

API_URL = "xxxxx/api/v1/prediction/aaa5cb74-d79f-49d9-a7c7-c71a1303b72a"

# Load the tokenizer for the model you are using
tokenizer = AutoTokenizer.from_pretrained("model_name_here")  # Replace with your model name

def query(payload):
    response = requests.post(API_URL, json=payload)
    return response.json()

def count_tokens(text):
    tokens = tokenizer.encode(text)
    return len(tokens)

# Example usage
question = "Hey, how are you?"
token_count = count_tokens(question)
output = query({"question": question})

print(f"Token count for the question: {token_count}")
print("API Output:", output)

Thank you very much. If I want to use this method to also calculate the tokens consumed for giving answers, will it significantly increase the time spent on the entire process?

[Yakiw]

hello @neehhaa06 Thank you very much. I didn’t find an API in flowise that can calculate the consumed tokens.

[HenryHengZJ]

there isnt a built in way for that at the moment, thanks @neehhaa06 for helping out!

[bendadaniel]

Hello, the code provided is not a solution, if there is some agent flow or some internal llm loop, it does not count these internal tokens. So this solution is not valid, it only counts base API input/output tokens.

However, this functionality is crucial to use Flowise in production, you need to know how many tokens you are using.

Are there any plans to implement it please? @HenryHengZJ