System.Threading.Tasks.TaskCanceledException: The operation was canceled

[maxiar]

Hi! I has this recurrent and random error in differents jobs (with diferents APIs, and no time pattern to reproduce it) but usually the process in the destination API is finishing ok, with no error logs, or exceptions. It think that it's a client disconnection..

System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.. ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token) at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async) at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Hangfire.HttpJob.Server.HttpJob.Run(HttpJobItem item, PerformContext context, List`1 logList, HttpJobItem parentJob) at Hangfire.HttpJob.Server.HttpJob.Excute(HttpJobItem item, String jobName, String queuename, Boolean isretry, PerformContext context)

I already see other issues closed like: https://github.com/yuzd/Hangfire.HttpJob/issues/217 https://github.com/yuzd/Hangfire.HttpJob/issues/67

but no found a solution, I was investigating during a few weeks without success to fix it:

The hangfireServer is hosted in a ASPNET Core (Net8) in IIS, and the jobs that it's calling to another ASPNET Core sites in another site in IIS.. all in the same server, but each application has a own site and own application pool. I'm not using HttpJob Agents.. but I think that it will not solve this error.

0. This error was found on diferent task and sites and duration:

1. I already configured the recommended settings in IIS for all app pools: On IIS Manager → Application Pools → Select the App Pool --> Mode: Always Running --> Idle Time-out (minutes) = 0 --> Regular Time Interval (minutes) = 0

2. I increment the timeout to 10800000ms (3hs) but the error occurres after 2min.. or 10min.. or 30min.. no time pattern or same job..

3. I setting a global http timeout (3hs) to hangfire configuration:

4. No error was found in the jobs configured and the process was end sucessfully after 10min, but the error in the hangfire dashboards appears 2 min later job started.

5. I activate the IIS logs to who more information and see that when this error was detected no exception or responde error was found, all are in a 200OK:

2025-02-13 16:02:02 127.0.0.1 POST /api/v1/scrapers - 7210 - 127.0.0.1 - - 200 0 0 17924 2025-02-13 16:03:44 127.0.0.1 POST /api/v1/scrapers - 7210 - 127.0.0.1 - - 200 0 0 103045

6. I check the IIS Server event viewer to see IIS issues or warning, restarting events, nothing found.

7. I'm using the latest vestion of your library and hangfire:

"Hangfire" Version="1.8.12"
"Hangfire.Console" Version="1.4.3"
"Hangfire.Console.Extensions" Version="1.0.5"
"Hangfire.HttpJob" Version="3.8.5"
"Hangfire.HttpJob.Client" Version="1.2.9"
"Hangfire.MySqlStorage" Version="2.0.3"
"Hangfire.PostgreSql" Version="1.20.5"
"Hangfire.SQLite" Version="1.4.2"

8. I call to job api with POSTMAN client and no error was detected and the postmain wait the responde for more that 48min in some test without errors:

9. No Resource issues on the server. (4 Cores and 32GB de RAM)

10. I can't remove the IIS or deploy the applications in another site like docker. It's a company requirement.. :(

For all those test I supossed that the problem are from the client side or hangfire or httjobs or any other server issue, but no found any evidence for it. I love hangfire and your httpjobs lib. I make all the project around this but this error is not good and marks some jobs as fail when the task internally was good, we has more that 100 scraping tasks and it is very annoying to re-check each of the failed tasks, to see if it really was a failure. Only for these reason I'm evaluating other tools like ELSA WORKFLOW or Temporal.IO but are more complex that I need.. ;( ;( ;( .. please help me to solve it. If you need we can make a google meet to investigate it or test something..

Can you think of any other tests or tests?

Beyond the error, it is correct, perhaps it is not correct to wait so long for a response when they are long tasks, (this error also happens with short-duration tasks as well), is there no other way to check if the task is finished correctly via pulling or signal without having to wait for it synchronously with a HTTP CALL?

Thank in advanced!

Regards!

[yuzd]

I know this issue,I tried to investigate, but didn't have any ideas. I'll think about it again. I'll discuss it with you if necessary.

[maxiar]

Good! , I was investigating the execution code and found it.. may be it can help us....

I think that the use of ConfigureAwait and GetAwaiter, and GetResult it is not a good practice and we should have dispose and concurrence issues

May be something like that was better, a little example....

` using System; using System.Net.Http; using System.Threading; using System.Threading.Tasks;

public class HttpService { private readonly HttpClient _httpClient;

public HttpService(HttpClient httpClient)
{
    _httpClient = httpClient;
}

public async Task<HttpResponseMessage> SendRequestAsync(RequestItem item, RequestContext context, ParentJob parentJob)
{
    using var cancelTokenSource = new CancellationTokenSource(TimeSpan.FromMilliseconds(item.Timeout));
    try
    {
        using var httpRequest = PrepareHttpRequestMessage(item, context, parentJob);
        return await _httpClient.SendAsync(httpRequest, cancelTokenSource.Token);
    }
    catch (TaskCanceledException ex) when (!cancelTokenSource.Token.IsCancellationRequested)
    {
        Console.WriteLine("⏳ Tiempo de espera agotado para la solicitud.");
        throw new TimeoutException("La solicitud HTTP ha excedido el tiempo de espera.", ex);
    }
    catch (HttpRequestException ex)
    {
        Console.WriteLine($"🚨 Error en la solicitud HTTP: {ex.Message}");
        throw;
    }
    catch (Exception ex)
    {
        Console.WriteLine($"🔥 Excepción inesperada: {ex.Message}");
        throw;
    }
}

private HttpRequestMessage PrepareHttpRequestMessage(RequestItem item, RequestContext context, ParentJob parentJob)
{
    return new HttpRequestMessage(HttpMethod.Get, item.Url); 
}

}

`

If you create or fix it using this practices in a branch... I can pull it, compile and use it as internal in my project to test if the a task was cancelled issue was solved, before to release a new nuget versions. bla bla bla....

and may be update to the latest version of hangfire lib.... wr are 4 versions ago...

REgards!