Azure SignalR Serverless function fails to send out a message when the object size becomes fairly large and reports error 413

[Arash-Sabet]

Describe the bug

We have a code similar to the following piece which sends a message to a userId by signalR:

[FunctionName("broadcast")]
		public static async Task SendMessageAsync(
			[HttpTrigger(AuthorizationLevel.Function, "post", Route = "broadcast/{group}/{userId}")]
			HttpRequestMessage req,
			string group,
			string userId,
			[SignalR(HubName = Negotitate.HubName)]IAsyncCollector<SignalRMessage> signalRMessages,
			ILogger logger)
		{
			string message = await req.UnzipAsync(); //This is an in-house library to unzip a request.
				await signalRMessages.AddAsync(
					new SignalRMessage
					{
						UserId = userId,
						GroupName = group,
						Target = "finished",
						Arguments = new[] { JsonConvert.DeserializeObject(message) } //deserialize the payload using Newtonsoft
					});
		}

The serverless signalR hosted on Azure is a free/development tier entity. Basically what this code does is that it obtains a compressed content (json payload) from the request, decompresses it and gets the SDK send the de-serialized object to the user in the group. This function works well when the payload yields a small object. When the SDK tries to submit an object whose payload is around 2.3 MB to Azure SignalR, it reports the following failure:

{"Response status code does not indicate success: 413 (Request Entity Too Large)."}

To Reproduce

Please follow the steps above. Simply supply a large payload exceeding 2.3MB and have it sent out via signalR on Azure.

Exceptions (if any)

at Microsoft.Azure.SignalR.Management.RestHubLifetimeManager.ThrowExceptionOnResponseFailure(Exception innerException, Nullable1 statusCode, String requestUri, String detail) at Microsoft.Azure.SignalR.Management.RestHubLifetimeManager.<CallRestApiAsync>d__23.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult() at Microsoft.Azure.WebJobs.Extensions.SignalRService.AzureSignalRClient.<SendToUser>d__13.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult() at Microsoft.Azure.WebJobs.Extensions.SignalRService.SignalRAsyncCollector1.<AddAsync>d__3.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult()

Further technical details

• Your Azure SignalR SDK version:
• Your Server ASPNETCORE version or Assembly version of Microsoft.AspNetCore.SignalR
• Your SignalR Client SDK version: n/a

X:>dotnet --info .NET Core SDK (reflecting any global.json): Version: 3.1.401 Commit: 5b6f5e5005

Runtime Environment: OS Name: Windows OS Version: 10.0.17763 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\3.1.401\

Host (useful for support): Version: 3.1.7 Commit: fcfdef8d6b

.NET Core SDKs installed: 3.1.102 [C:\Program Files\dotnet\sdk] 3.1.202 [C:\Program Files\dotnet\sdk] 3.1.401 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed: Microsoft.AspNetCore.All 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 3.1.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

[wanlwanl]

Could you try Persisent mode? Change the settings from Transient to Persisent.

Trasient will send http request to call the REST API, which has the limit body size of 1M. But in Persisent mode, websocket connections will be established to the service, the large message will be split into small pieces and send to ASRS.

But I suggest to send small messages to SignalR service, if the message is too large, the speed might be slow, which might cause the connection drop.

[Arash-Sabet]

@wanlwanl Thanks for the reply. I made the change and now I started getting the following exception:

"The collection type 'Newtonsoft.Json.Linq.JObject' is not supported."

Is there a documentation somewhere online to elaborate all such configurations clearly?

[wanlwanl]

@Arash-Sabet Could you try our sample with Persisent mode? Once you run the sample, open the html to send messages. I can't reproduce the "The collection type 'Newtonsoft.Json.Linq.JObject' is not supported." issue.

Is it possible to provide a minimal reproducible repo to investigate?

Our load balancer has the limit, we are going to update our documentation, thank you for pointing out!

[Arash-Sabet]

@wanlwanl I changed the code to useSystem.Text.Json.JsonSerializer.Deserialize<T>(string payload) instead of Newtonsoft to get rid of the error message of "The collection type 'Newtonsoft.Json.Linq.JObject' is not supported." but that caused worse issues.

This is now what I get from signalR:

I think you need to submit an instance of object "B" in the following example by POST action to the HTTP triggered function to be able to reproduce the error that I reported:

public class A
{
    public int Id { get; set; }
    public string Name { get; set; }
}

public class B
{
    public int Code { get; set; }
    public A[] MyArray { get; set; }
}

Try to instantiate class B, populate array MyArray in the instance and have signalR send the instance of B.

This is how you'd populate SignalRMessage argument and assume that the concrete types of A and B are unavailable to the http triggered function:

Arguments = new[] { System.Text.Json.JsonSerializer.Deserialize<dynamic>(message) }

Apparently System.Text.Json.Deserializer is unable to preserve object graph when there's a hierarchy of nested objects.

[wanlwanl]

Could you try the sample (in wanl/issue1001 branch, which uses the class B) https://github.com/Azure/azure-functions-signalrservice-extension/tree/wanl/issue1001/samples/simple-chat ?

I can't reproduce, and I am able to use Newtonsoft.Json.JsonConvert.

[Arash-Sabet]

@wanlwanl I will get back to you later on with further details. We'll try it in a couple of weeks.

[Arash-Sabet]

@wanlwanl I used the sample application, modified it slightly to be able to send our payload and it worked fine. Our function app is a v3 with the following project configuration:

<TargetFramework>netcoreapp3.1</TargetFramework>
<AzureFunctionsVersion>v3</AzureFunctionsVersion>
<_FunctionsSkipCleanOutput>true</_FunctionsSkipCleanOutput>

I'm not still sure why I get that error message in our project but not in the sample chat application.

[wanlwanl]

Glad to hear you solve the problem! Looks like the function will remove some new packages by default. They use https://github.com/Azure/azure-functions-host/issues/5894 to check the fix.

[Arash-Sabet]

@wanlwanl Well, the issue still persists.

[wanlwanl]

Well, if you still need to use your project config... Could you share the minimal reproducible project to me? Let me try to fix the issue.

[Arash-Sabet]

Preparing a minimal project takes quite a bit of time and I need to schedule it. There is some business logic that must be eliminated from the project prior to sharing it. I'll keep you posted.

[Arash-Sabet]

@wanlwanl Please see the repo that I created here: https://github.com/Arash-Sabet/AzureSignalRFunctionIssue

You need to supply your own signalR URL and App insights' key.

[wanlwanl]

@Arash-Sabet The aspnetcore 3.x System.Text.Json the default for SignalR and remove Newtonsoft from shared framework, our SDK depends in the aspnetcore signalr sdk, so you need to change to use System.Text.Json if you want to keep the target framework.

About your issue

Apparently System.Text.Json.Deserializer is unable to preserve object graph when there's a hierarchy of nested objects.

I make a test, I found nested json (based on your samplepayload.json) is fine to parse, e.g. parse the category id and keys:

var obj = System.Text.Json.JsonDocument.Parse(message);
var catIds = obj.RootElement.GetProperty("Categories").EnumerateArray().Select(c => c.GetProperty("CategoryId").GetInt32());
var keysToStore = obj.RootElement.GetProperty("OtherCommands").GetProperty("KeysToStore").GetString();				

There's the migration guide from Newtonsoft.Json to System.Text.Json.

[Arash-Sabet]

@wanlwanl Just trying to reiterate and clarify my previous comments a bit:

I had used System.Text.Json in singalR server and got it send out the object. The client application whose code is per the following snippet received the object and printed the message per the screenshot after the code snippet.

connection.On<object>("CommandCompleted", (message) =>
{
	Console.WriteLine($"Received the following packet at command completed:\r\n {message}");
});

The only workaround that worked for us was to serialize the object into its JSON payload string, send the string to the singalR client app and have it deserialize the payload.

[Arash-Sabet]

@wanlwanl The client application does not know anything about the nature of YourClass. The mainstream application is an angular UI app which does not have the concrete definition of the C# class.

[wanlwanl]

when using Newtonsoft will serialize into a JObject which isn't supported by System.Text.Json. And dynamic with System.Text.Json is not supported yet https://github.com/dotnet/runtime/issues/29690.

You either need to use concrete types in their C# Functions app or serialize and deserialize to/from strings manually.

The js client always get json string, so I think a concrete type can work. But you can keep manually serialize and deserialize to/from strings as a workaround.

[Arash-Sabet]

@wanlwanl Using a concrete class in the client side is impossible because there are a lot of objects with different graphs reported by signalR. The only common areas in those objects are a few properties.

[Adrian10988]

Could you try Persisent mode? Change the settings from Transient to Persisent.

Trasient will send http request to call the REST API, which has the limit body size of 1M. But in Persisent mode, websocket connections will be established to the service, the large message will be split into small pieces and send to ASRS.

But I suggest to send small messages to SignalR service, if the message is too large, the speed might be slow, which might cause the connection drop.

Apologies for the necromancing but it was pretty difficult for me to find this thread and solve the same issue Arash was having. Thank you wanlwanl for the fix, I just wanted to point out the value Persisent is misspelled. I span my head around for a while trying to figure out why I was still receiving 413 from ASR and it turned out I just needed to spell it correctly. I guess copy paste is not always reliable :D

Persistent