PullRequestInc
No Token Usage Information when Streaming
Hello, when I use the CompletionStreamWithEngine function I notice that all token values are marked as 0. Is it safe to assume that every message that is streamed is a single token?
Hi @abdulrabbani00 , I don't believe that it streams only one token at a time from the API, but I'm not 100% sure. Although it is possible that openai is not including these token counts in the response when using streaming.
I don't have the time to investigate currently, but I would suggest potentially trying out the openai python library and see if they give you tokens there or not. if not then they probably aren't available in the API response.
Based on this post in the OpenAI community forum, this is a limitation at the API level and not the library. I also verified this by testing the python API and confirmed it doesn't have the usage info there either.
That same message mentions that you could potentially get usage information to be populated in the stream API for your OpenAI org if you contact them on https://help.openai.com/en/ to get the feature enabled, so that's an option. Although, it seems like they are implying that the way it is implemented could break existing integrations so not sure if it is backward compatible. E.g., turning on the feature on your account might break the implementation in this library. I don't have it turned on on my OpenAI account so don't have a way to test it to be sure...
