How to incrementally apply coalescing behavior when streaming chat

[ericstj]

The user problem is how to represent streaming updates without re-implementing the coalescing behavior we have in ToChatResponse https://github.com/dotnet/extensions/blob/192782ef64ab19202f054e32e62fc7c857a33d23/src/Libraries/Microsoft.Extensions.AI.Abstractions/ChatCompletion/ChatResponseExtensions.cs#L126-L136

Our template does a very basic thing of just concatenating all text https://github.com/dotnet/extensions/blob/192782ef64ab19202f054e32e62fc7c857a33d23/src/ProjectTemplates/Microsoft.Extensions.AI.Templates/src/ChatWithCustomData/ChatWithCustomData-CSharp.Web/Components/Pages/Chat/Chat.razor#L74

Here's a sample of how I adapt that for also handling images -- https://github.com/ericstj/imageGeneratorSample/blob/bed6c588a84489a9b652613a8e8f2ac24a20c2f2/imageGeneratorSample.Web/Components/Pages/Chat/Chat.razor#L98-L169

Related discussion: https://github.com/dotnet/extensions/pull/6749#discussion_r2479552384 Here I proposed ApplyUpdate API that would allow folks to incrementally coalesce as content arrives.

While this satisfies the need, it does have the risk of excessive complexity - given N updates, each time iterating over all content to try to coalesce, and again to shift data over removed items - the existing implementation of ToChatResponse tries to avoid this complexity but incremental APIs would probably have it effectively when called for each update. Even though the result of incremental coalescing would be smaller - especially smaller in the case of contiguous items - there's still a risk in worst case.

It feels like a sorting problem. Perhaps that's how to solve it. If we had a data structure to store updates in that behaved like a sorted list with all the coalescing rules, and exposed ChatMessages.

@stephentoub