[Question] When Azure will sync OpenAI ChatCompletion stream_options feature ?

[SKYhuangjing]

API Spec link

https://github.com/Azure/azure-rest-api-specs/blob/main/specification/cognitiveservices/data-plane/AzureOpenAI/inference/preview/2024-04-01-preview/inference.json

API Spec version

2024-04-01-preview

Question/Query

OpenAI ChatCompletion API has add stream_options {include_usage} for stream get usage, it will help developer to statistics the api cost, it's very good for us, so when azure will sync it?

Environment

REST API invoke it

[zzhxiaofeng]

@YunsongB Please help have a look, thank you.

[dq7532183]

+1

[dbg-raghulkrishna]

+1

[dzmitryi-h]

+1

[janaka]

Feat request here https://github.com/Azure/azure-sdk-for-net/issues/44237

[SKYhuangjing]

any update? @zzhxiaofeng @YunsongB

[elabbarw]

+1

[supuwoerc]

+1+1+1

[pedro-n-rocha]

+1

[janaka]

I'm guessing this all now comes along with the new OpenAI client that pulled in v2.0.x

[dq7532183]

when?

[janaka]

https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/openai/Azure.AI.OpenAI/CHANGELOG.md#200-beta3-unreleased

[sandangel]

Hi, I tried with 2024-05-01-preview but still get the error. 400 bad request. I think it needs to be enabled on the server side, not client.

[sergiogarlez]

+1

[janaka]

Even in the new OpenAI lib (That Azure OpenAI v2.0.* depends on), this isn't implemented yet.

See https://github.com/openai/openai-dotnet/issues/103#issuecomment-2201442467

[mfournioux]

Are there any update on this issue?

[Funnyguy77]

Bump. Hoping to see this implemented.

[xin9le]

It appears that the OpenAI API is prepared to accommodate token usage reporting, as evidenced by the IncludeUsage = true setting in the Azure.AI.OpenAI v2 SDK implementation. While the specific obstacles preventing REST API support remain unclear, there is likely a substantial number of developers who desire to obtain token usage information via the Streaming API. Given that the current situation necessitates manual calculation of token usage, which is a considerable inconvenience, we earnestly hope for prompt support of this feature.

[Funnyguy77]

Given that the current situation necessitates manual calculation of token usage, which is a considerable inconvenience, we earnestly hope for prompt support of this feature.

I would also like to note that manual token calculation is often not accurate, in addition to being a large inconvenience. Making this feature even more desirable to developers, like myself.

[DrShushen]

This would be very useful - as @Funnyguy77 said, counting tokens manually is very error prone, and not maintainable.

[janaka]

It would be great if the Azure OpenAI team could give customers some indication as to when this feature will be released.

In general you really need to do a better job communicating the policy for maintaining feature parity with OpenAI feature releases. Right not this is one of several inference features that many are waiting on with absolutely no info on the Azure OpenAI teams intentions.

[MartialBE]

When I try to request gpt-4o-mini, stream_options is valid for me but not for other models. Why is this inconsistency happening?

[SKYhuangjing]

https://github.com/Azure/azure-rest-api-specs/pull/30087 it seems have some update, but not nofity us.

[SKYhuangjing]

When I try to request gpt-4o-mini, stream_options is valid for me but not for other models. Why is this inconsistency happening?

Compare eastus & northcentralus API Result (api_version = 2024-07-01-preview)

• eastus has support stream_options
• northcentralus has not support it hard to see why...

[janaka]

When I try to request gpt-4o-mini, stream_options is valid for me but not for other models. Why is this inconsistency happening?

Compare eastus & northcentralus API Result (api_version = 2024-07-01-preview)

• eastus has support stream_options
• northcentralus has not support it hard to see why...

iirc for another feature (tool_choice=required) somebody reported similar that it's avail in one region but not another. I think with that it was avail in one of the euro region but not in eastus.

[SoulBlaze4j]

How is it going? Can't imagine it has been a long long time and Azure still cannot catch up with this feature even in the preview API version! :))

[MartialBE]

I'm currently in the swedencentral/northcentralus/japaneast regions, and stream_options is working correctly. Is it fully available now? api-version: 2024-07-01-preview

[SKYhuangjing]

has been released https://learn.microsoft.com/zh-cn/azure/ai-services/openai/api-version-deprecation#changes-between-2024-09-01-preview-and-2024-08-01-preview

[SoulBlaze4j]

has been released learn.microsoft.com/zh-cn/azure/ai-services/openai/api-version-deprecation#changes-between-2024-09-01-preview-and-2024-08-01-preview

GPT-4 vision-preview still does not support this feature, I tested in westus region and it returned { "error": { "message": "1 validation error for Request\nbody -> stream_options\n extra fields not permitted (type=value_error.extra)", "type": "invalid_request_error", "param": null, "code": null } }

[kglmcodes]

+1