Huge latency for (recurring) jobs

[GabrielGorgan]

Hello everyone,

I have few recurring jobs and they were working perfectly fine but the last one had a really big latency:

Web app specs:

• .NET 5 (Angular front-end)
• SQL Server
• App hosted on IIS on an Azure VM

Could you please advise what's causing this huge delay?

[odinserj]

Sometimes this is possible due to unsynchronised clocks between servers. Please ensure all of them show the same time.

[GabrielGorgan]

How can I ensure the clocks are synchronized? For example, when using AddOrUpdate I should also provide TimeZoneInfo (UTC or Local)?

Minor update regarding job status: Today, the job had a normal latency (ms).

[odinserj]

Sorry for such a huge delay. Hangfire uses UTC time internally for storing all the dates, so timezone information shouldn't cause this. Clocks are synchronised when all the machines show the same UTC time, and using some kind of time server synchronisation is preferred.

If everything is fine with time on your machines, another possible reason is that background processing server was inactive for that time – for example process was stopped or not running after machine restart, idle timeout in IIS or connectivity problems with your job storage (in this case logs should contain the corresponding exceptions).

If your process was up and running, then it is possible for different other issues to prevent processing a job – CLR's thread pool starvation, out of memory issues, etc. In this case it's better to look what's running on your threads by using a tool like stdump I wrote to have better diagnostics for cases where everything is stuck.