semantic-kernel icon indicating copy to clipboard operation
semantic-kernel copied to clipboard

Token usage stats from responses should be returned to the client.

Open SOE-YoungS opened this issue 2 years ago • 1 comments

OpenAI / Azure OpenAI both return token stats with responses.

These are useful for tracking usage etc... on a case by case basis for things like "inter-departmental billing" etc...

Currently SK only supports returning the completion (string), meaning token usage is not accessible to the end user & unavailable for future evaluation / tracking.

SK, should return an object containing token stats & completion string.

Example use case(s):

  1. A large enterprise (parent company pays the bills) with subsidiaries (AI implementation end users). The parent will want to track token usage by each subsidiary to ensure cross charging for usage is correct.
  2. Stored as a statistic in an end users account. This allows the end user to see & be aware of token usage over time VS individual queries.

SOE-YoungS avatar Apr 24 '23 15:04 SOE-YoungS

@SOE-YoungS , great idea, we will take a look at getting this added.

evchaki avatar Apr 27 '23 20:04 evchaki