deepset-ai
Add Token Usage Tracking for Haystack PromptNode
Is your feature request related to a problem? Please describe.
As a user of the Haystack PromptNode and Agent, it is currently difficult to gather insightful data about the tokens used in my deployed application. I'm seeking a way to track the tokens used in prompting and those used in generating responses. This data is valuable for understanding the efficiency and effectiveness of my system and helping me optimize it for better cost and results.
Describe the solution you'd like
I'd love to see a new feature that allows an easy programmatic or configuration switch to initiate this tracking of tokens. The tracking should be comprehensive, capturing every prompt and the generated text that corresponds to it. For ease of data analysis and to track trends over time, it would also be beneficial if each entry contained a timestamp.
In an ideal world, the feature would allow for:
Programmatic initiation of token tracking. Logging of tokens used in both prompting and response generation. Each log entry contains a timestamp. The ability to easily export this data for further analysis.
Describe alternatives you've considered
Unfortunately, there are no existing alternatives or workaround solutions for this kind of feature within Haystack or elsewhere. Without this feature, users are left in the dark about detailed token usage in their applications.
Additional context
Having access to such statistics can be of immense help to Haystack users. It could help identify areas where token usage might be more than necessary or where optimizations can be made. It would also be interesting to observe the token usage patterns over time. This feature could be an important tool for enhancing the performance of Haystack-based applications.
:+1:
This feature would be very useful!
While experimenting with @bilgeyucel on prompt lemmatization to save tokens, we discovered that in some cases (OpenAI), the token usage is returned in the API response, while other providers (Cohere) don't provide this information out of the box.
The Solution Engineering team would also find this to be very useful! We've had people request being able to track token consumption over time on their hosted pipelines. We currently have to do this after the fact by downloading search history and running a tokenizer over this search history.
