BUG: include_usage for streaming doesn't work due to an SDK client bug

[janaka]

Service

Azure OpenAI

Describe the bug

This feature seems to be switched on to true by default in the OpenAI client. However, the StreamingChatCompletionUpdate.Usage property is always null.

Before CompleteChatStreamingAsync(messages, options) is called when I inspect the options object, Stream=null and StreamOptions.IncludeUsage=true. However, immediately after CompleteChatStreamingAsync(messages, options) is called when I inspect the options object, Stream=true and SteamOptions is null.

Inspecting the sdk source code, this appears to be a bug in how the internal constructor code is implemented.

Azure client version: 2.1.0 Beta 1
OpenAI Client version: implicit (>= 2.1.0-beta.1)
OpenAI Inference API version: AzureOpenAIClientOptions.ServiceVersion.V2024_08_01_Preview
OpenAI API provider : Azure OpenAI
Model name: GPT-4o 08-06

Just before calling CompleteChatStreamingAsync:

Just after calling CompleteChatStreamingAsync:

Steps to reproduce

Use the v2.1.* dotnet client to make a streaming call this should return usage. But it doesn't.

Making a Postman request, with the correct options for include usage, to the same deployment does return token usage counts.

Code snippets

No response

OS

macOS

.NET version

netstandard 2.0

Library version

2.1.0-beta1

[joseharriaga]

Thank you for reaching out, @janaka ! stream_options is not "officially" supported in Azure OpenAI until version 2024_09_01-preview of the service (it's confusing because it was previously enabled partially in some regions but not all of them). We're working on a release to add support for this version soon.

[janaka]

The problem is not with the inference API from what I can tell though. It's a dotnet library issue I'm seeing. The request going out does not have include_usage = true

[janaka]

@joseharriaga assuming by 2024_09_01-preview you are referring to the inference rest API version, I've just tried switching to that (using reflection given the enum on has 08-01-preview and 10-01-preview). There request payload still doesn't have include_usage when I inspect the http request going out. Please let me know if I'm mis understanting what you were referring to by 2024_09_01-preview

In any case I'm not expecting the REST API version changing to adjust the client lib behaviour. I don't see where this option is switch off in the client (is it?). So I expect the http request going out to have "stream_options":{ "include_usage": true }.

[joseharriaga]

@janaka Your understanding is correct! Because stream_options was in this limbo where it only worked "unofficially" in a few regions until now, we opted for removing include_usage: true inside the library because it would cause chat completions to crash in any regions where it wasn't available. Now that it is "officially" supported everywhere with version 2024-09-01-preview, we are changing the library to allow include_usage: true to go through as expected.

[janaka]

@joseharriaga that's awesome! will that be in 2.1.0-beta.2 ? will it be soon like in the next couple weeks, this quarter, or next quarter, or beyond next quarter?

[joseharriaga]

@janaka Yes, we're aiming to include it in 2.1.0-beta.2, which we expect to release between this week and next week.

[joseharriaga]

@janaka This is now supported in version 2.1.0-beta.2, which we released yesterday.