Chat completion stream returns empty total usage

[JosePina98]

Confirm this is a Node library issue and not an underlying OpenAI API issue

• [X] This is an issue with the Node library

Describe the bug

Using openai.beta.chat.completions.stream() and then calling totalUsage function returns the object with all values set to zero.

To Reproduce

1. Call the chat completions API with response stream
2. Log the result of totalUsage function
3. Check that every value is zero

Code snippets

const response = openai.beta.chat.completions.stream({
    messages: [
        {
            role: "user",
            content: "hello"
        }
    ],
    model: "gpt-3.5-turbo-1106",
    response_format: { type: "text" },
    max_tokens: 300,
    n: 1,
    temperature: 1.2,
    stream: true
});

console.log(await response.totalUsage())

OS

Ubuntu

Node version

Node v16.13.0

Library version

open v4.19.0

[clickeo-dev]

I can confirm that this issue still exist. @JosePina98 did you find any workaround on this?

[JosePina98]

For now I have created this Typescript function to estimate the tokens used in each call. According to the tests I've done, it makes a rough estimate, but on the high end.

const tokenEstimation = (prompt: string, output: string): {
    prompt_tokens: number,
    completion_tokens: number,
    total_tokens: number
} => {

    const tokenLength = 3.8;

    const promptTokens = Math.ceil(prompt.length / tokenLength);
    const completionTokens = Math.ceil(output.length / tokenLength);
    return {
        prompt_tokens: promptTokens,
        completion_tokens: completionTokens,
        total_tokens: promptTokens + completionTokens
    }
}

[clickeo-dev]

For now I have created this Typescript function to estimate the tokens used in each call. According to the tests I've done, it makes a rough estimate, but on the high end.

const tokenEstimation = (prompt: string, output: string): {
    prompt_tokens: number,
    completion_tokens: number,
    total_tokens: number
} => {

    const tokenLength = 3.8;

    const promptTokens = Math.ceil(prompt.length / tokenLength);
    const completionTokens = Math.ceil(output.length / tokenLength);
    return {
        prompt_tokens: promptTokens,
        completion_tokens: completionTokens,
        total_tokens: promptTokens + completionTokens
    }
}

I do kinda same. Using the tokenizer package since the package doesn't support new models I hard coded 'gpt-4' as model. I assume that they are all using cl100k_base encoding model under the hood.

 // Calculate Token Sizes
const inputTokenSize = tokenizer.encodeChat(messages, "gpt-4").length;
const outputTokenSize = tokenizer.encodeChat([{role: "assistant", content: output,},], "gpt-4").length;

[rattrayalex]

Sorry about this, usage events are not sent in streaming just yet; OpenAI hopes to add that to the API soon. Once that happens, this function will return numbers as you'd expect.

It was arguably a mistake to include totalUsage() on that interface before it was ready in the API, sorry for the confusion.

[yauri-io]

Hitting this issue as well. I see this issue's status is Closed, but I couldn't see any new release with the fix. Any ETA?

[rattrayalex]

cc @athyuttamre

[athyuttamre]

Hi folks, we pre-emptively added the totalUsage() function but unfortunately have not released support in the API just yet. We plan to do so early next year. Sorry for the confusion, but feel free to subscribe to this GitHub issue and we'll post here when it is launched.

[talos]

Glad to hear it! Would it be possible to re-open the ticket to make it clearer when this has been resolved?

[jay-pulsifi]

Hi team, any ETA when this ticket will be resolved?

[aar2dee2]

following this for updates

P.S. thanks for the stream events - have made it easy to work with db calls when streaming completions!

[Mickeygogo]

When are you going to put this issue on to resolution?

[nai-kon]

I am really looking forward to usage support on stream mode.

[donprasetiyo]

The Python one has been shipped. Is there an estimate for the node?

[decline]

I'm using version 4.51.0 and I just found out there is a stream_options property that can contain include_usage: true, example:

 const stream = this.openAi.beta.chat.completions.stream({
      model: 'gpt-3.5-turbo',
      messages: [{ role: 'user', content: 'Say something funny' }],
      stream: true,
      max_tokens: 5,
      stream_options: {
        include_usage: true
      }
    });

When using it like this, you are able to either use the stream.totalUsage() promise or the stream.on('totalUsage', usage => {}) callback. In both cases, you will receive:

{ completion_tokens: 5, prompt_tokens: 10, total_tokens: 15 }

Not sure if this feature was added silently, or nobody noticed?

[nai-kon]

@decline

Thank you for the information. It seems the usage option on streaming was supported on May 6th. Here is the change log.

https://platform.openai.com/docs/changelog/may-6th-2024

Now I added usage fee notation function on my app. It works perfect.

@JosePina98 Can we close this issue?

[Mickeygogo]

yes

要买直升机的Mickey @.***

 

------------------ 原始邮件 ------------------ 发件人: "openai/openai-node" @.>; 发送时间: 2024年7月22日(星期一) 中午1:52 @.>; @.@.>; 主题: Re: [openai/openai-node] Chat completion stream returns empty total usage (Issue #506)

It seems the usage option on streaming was supported on May 6th. Here is a release note.

https://platform.openai.com/docs/changelog/may-6th-2024 image.png (view on web)

Can we close this issue?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[rattrayalex]

As @decline noted, you can now access total usage data by passing stream_options: { include_usage: true }.