Owin throwing TaskCanceledException when selfhosted application receives more than 3k concurrent requests

[arpitha-kp]

Hi, I'm new to OWIN, if I get some help on this issue, I really appreciate it. I'm trying to do scalability testing of ASP.net Web API which is self-hosted as a windows service. It works fine for concurrent API requests up to 70 (each request internally further creates around 50 requests approximately - out of which 24 are concurrent, once responses are received again 18 calls will be concurrent and the rest all are sequential). But, When I started to run 100 concurrent requests it starts to fail for a few requests and it threw the following error:-

"System.Threading.Tasks.TaskCanceledException: A task was canceled. at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Owin.HttpMessageHandlerAdapter.<SendResponseContentAsync>d__20.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Owin.HttpMessageHandlerAdapter.<InvokeCore>d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()"

When I look into the git repository for owin's SendResponseContentAsync Implementation (https://github.com/aspnet/AspNetWebStack/blob/cd0186268be4679be68295e42428d8b18d8e09a1/src/System.Web.Http.Owin/HttpMessageHandlerAdapter.cs#L567), I'm assuming it's throwing an exception at Ln578, which inturn returning a Task with canceled state at Ln596 via AbortResponseAsync() method. I'm guessing it's happening due to resource exhaust.

Is there any OWIN configuration available that I can use to handle multiple concurrent requests, Can I get any help, please?

[Tratcher]

(Note new projects should be using AspNetCore instead of OWIN).

Are you able to access the logs to see what the original exception might be?

[Tratcher]

Duplicate of https://github.com/aspnet/AspNetWebStack/issues/361

[arpitha-kp]

(Note new projects should be using AspNetCore instead of OWIN).

Are you able to access the logs to see what the original exception might be?

Thank you for your response and suggestion

After implementing System.Web.Http.ExceptionHandling.ExceptionLogger, I'm able to access the inner exception and it logged the following error:-

System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException ---> System.Net.HttpListenerException: The I/O operation has been aborted because of either a thread exit or an application request at System.Net.HttpResponseStream.EndWrite(IAsyncResult asyncResult) at Microsoft.Owin.Host.HttpListener.RequestProcessing.ExceptionFilterStream.EndWrite(IAsyncResult asyncResult) --- End of inner exception stack trace --- at Microsoft.Owin.Host.HttpListener.RequestProcessing.ExceptionFilterStream.EndWrite(IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization) --- End of inner exception stack trace --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Owin.HttpMessageHandlerAdapter.<SendResponseContentAsync>d__20.MoveNext()

[Tratcher]

That looks like the request is being cancelled from another thread. Maybe there's a timeout being applied? If your system is CPU saturated then you may not be able to process this many concurrent requests and they start to fail. You may need to lower your max concurrent request limit in the server. https://github.com/aspnet/AspNetKatana/blob/77497960c0adafcf2adaed1dc008f020595b28ab/src/Microsoft.Owin.Host.HttpListener/OwinHttpListener.cs#L81

[arpitha-kp]

Thank you. I'll check as you suggested.