.Net: Bug: Semantic Kernel inflates token usage of input prompt by 20% due to Unicode escaping and URL encoding of the prompt

[Evyatar108]

trafficstars

Describe the bug the semantic kernel package in c# is inflating the usage tokens by around 20% for my prompts because it escapes Unicode characters and URL encodes the string of the content field of the messages array.

I tested it on a prompt that has 49k input tokens based on the metadata information from the Azure OpenAI api, and when using semantic kernel it inflates the input tokens to 63k

It's very easy to see the prompt differences when it has JSON in it

Platform

• OS: Windows
• IDE: Visual Studio
• Language: C#
• Source: NuGet package version 1.15

[blinchi]

And it's not just the consumption of tokens, it's that it costs more for the LLM to process it. When receiving encrypted data, the responses are sent to you encoded, when it should not be that way. In my case using Gemini.

[atiq-bs23]

@markwallace-microsoft can you please check #7052

[markwallace-microsoft]

Fixed via https://github.com/microsoft/semantic-kernel/pull/7098

[blinchi]

This is not fixed in dotnet version 1.16.2. When the prompt is generated it still sends it with escaped characters.

This occurs when a prompt is passed:

[sonphnt]

Do we have a right way to see a prompt is actually sent to the azure open ai instead of checking chat history message? I have the same issue here but with function calling, somehow uses it a lot of tokens even though the function returns a very simple data and end up with thousands of tokens used

[dmytrostruk]

Do we have a right way to see a prompt is actually sent to the azure open ai instead of checking chat history message? I have the same issue here but with function calling, somehow uses it a lot of tokens even though the function returns a very simple data and end up with thousands of tokens used

@sonphnt Yes, if you enable LogLevel.Trace in your application, you will be able to see the prompt that is sent to AI connector, like on the screenshot in previous comment.

[alexandreakdeniz-soc]

Any update about it ? Looking at the sources it seems it comes from the way the chat history is logged using JsonSerializer without any option.

Locally I can reproduce by serializing a TextContent, then I can fix with this options: JsonSerializerOptions options = new () { Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping };

Perhaps being able to pass down this option would be a way to fix this. It is quite destabilizing when looking at the logs. Moreover, I do not feel entirely confident on how the data is really sent at the end.