Change hangfire initialization is causing different behavior on graceful shutdown

[joaozacharias]

Hello, I'm updating the hangfire version as i checked the deprecated message when i use 'app.UseHangfireServer()' method. After did this and start to use AddHangfireServer() with service collection instead UseHangfireServer() with application builder, the graceful shutdown doesn't work anymore, cancellation token is not triggered and the server/workers is not going down after receive the SIGTERM.

I clone hangfire application and made the change bellow and now seems to work.

Could you please check if this change is valid? And if it's valid, please apply.

[odinserj]

Hm, according to the docs the StopAsync method should also be called during the shutdown, so I except that these features duplicate each other. If not, we should conduct an investigation and check .NET Core's source code to see the difference, e.g. when and how ApplicationStopping/StopAsync methods are called.

[joaozacharias]

Hi, I got you. During my tests the StopAsync() is not called as soon the application receive the SIGTERM and then the cancellationToken is not triggered and workers/serves are active. When i used this another initalization that register the actions in application lifetime everything went well. I'm using .Net Core 3.1, please if you could do the investigation. Thanks!

[odinserj]

Need to check this also with .NET 5.0 and .NET 6.0 to understand how it works and whether behavior is different between different features before deciding what to do. Because if everything works fine in one of next versions, then it can be probably just a bug in .NET itself (already happened a lot of times), but in this case workaround in Hangfire itself can be considered to avoid problems when possible.

[odinserj]

Will continue investigation a bit later, was struggling to install .NET 5.0 on Ubuntu 22.04 box, then struggling to install supported OpenSSL 1.X, now struggling to make SSL working to connect to job storage 🤗

[joaozacharias]

I'm still doing some tests and i think i found the problem, i have another "IHostedService" in my application with a Thread.Sleep in the method hostApplicationLifetime.ApplicationStopping.Register(), when i remove this service from DI the cancellationToken is triggered but the application goes down without time to have a graceful shutdown, this occurs when i use AddHangfireServer(), when i use app.UseHangfireServer() everything works fine.

[joaozacharias]

When i try to use the options below, the cancelation token is not triggered. options.ShutdownTimeout = TimeSpan.FromSeconds(15); options.StopTimeout = TimeSpan.FromSeconds(10); options.ServerTimeout = TimeSpan.FromSeconds(10);

[odinserj]

@joaozacharias can you also try to increase the .NET Core's HostOptions.ShutdownTimeout value as written in this post at least to 20 seconds to be sure the host doesn't affect the behavior?

[odinserj]

i have another "IHostedService" in my application with a Thread.Sleep

But this can be the reason and if .NET Core allows one background service to block another ones during shutdown, then probably using ApplicationStopping all the times is the best idea to have a consistent behavior.

[joaozacharias]

@joaozacharias can you also try to increase the .NET Core's HostOptions.ShutdownTimeout value as written in this post at least to 20 seconds to be sure the host doesn't affect the behavior?

Yes, i did. This "ShutdownTime" value is 90 seconds.