openai
openai copied to clipboard
Add Cached Tokens to UsageResponse
Openai Api has added prompt caching : https://openai.com/index/api-prompt-caching/ This has added new property to the usage response returned this contain the number of tokens that where retrieved from the cache. Here is the example from the docs usage: { total_tokens: 2306, prompt_tokens: 2006, completion_tokens: 300,
prompt_tokens_details: { cached_tokens: 1920, audio_tokens: 0, }, completion_tokens_details: { reasoning_tokens: 0, audio_tokens: 0, } }
I have added a new class for prompt_tokens_details with property cached_tokens . The prompt_tokens_details has been added as a property of usage.