coravel

coravel copied to clipboard

Describe the bug

I think _scheduler.IsRunning is never getting set to false, that's why it just keeps looping in the while loop. Because when I try to stop my service, I see the warning log and windows service's status gets stuck in "Stopping" status

Affected Coravel Feature I think _scheduler.IsRunning is not getting set to false properly. I have jobs that run every minute maybe that's why

Expected behaviour If I need to stop my windows service than I should be able to, I think we can either configure Coravel to skip StopAsync method all together if I don't want to, or some sort of escape from while loop if it's been waiting a lot.

But most of the time I need an immediate stop if I want to stop it, so I think I'd choose a configuration to skip StopAsync method all together

This is where implementing Coravel.Invocable.ICancellableInvocable on an invocable class can help you (see docs for more). If there's a loop, for example, where you're making API or DB requests to do stuff, then you can check the cancellation token after each loop iteration (or even more fine-grained) to see if the app is trying to close, and then manually skip the rest of the method.

I don't even think ICancellableInvocable's cancellation token even do anything in this case. I'll explain my discovery.

I think what happens is when still there's a Coravel task processing(Scheduler.RunWorkersAt is processing) and at the same time windows service stop gets called, it stops the processing requests first and then it calls IHostedService.StopAsync. And then since processing requests are stopped _schedulerIterationsActiveCount field is stuck in a number that is greater than 0. That's why it just loops and loops inside of the while loop forever because it won't ever get back to 0 because processing requests were terminated.

To back up what I'm saying, this is the code of .net core

If you look at line 329 you'll see Server.StopAsync method get's called before IHostedService.StopAsync. And what does Server.StopAsync do is it "Stops processing requests and shut down the server, gracefully if possible."

https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.hosting.server.iserver.stopasync?view=aspnetcore-5.0

So before it even calls your IHostedService.StopAsync at line 335, it probably already terminated the other processing requests that's why it's stuck in the while loop

That explanation doesn't make much sense to me. Surely IServer.StopAsync just means the server will stop processing incoming requests, not somehow terminate all running tasks? The only reason _schedulerIterationsActiveCount could stay above 0 is if one of the tasks in RunWorkersAt hasn't completed yet.

Are still seeing this behavior when using the ICancellableInvocable?

I haven't checked IServer.StopAsync 's implementation yet, checking it right now.

I'm pretty sure that processing requests getting terminated that's why _schedulerIterationsActiveCount gets stuck above 0. I have logs all over the place in application and I also added a log inside the while loop like this:

Whenever trying to stop the service all the other logs in the application stops, web server becomes unavailable, and the only thing I see is coravel preventing service from stopping inside of the while loop.

Last time I tried stopping it was an hour ago, so it's been looping for an hour as you can see:

and there are no logs in between. I have no process that takes more than hour, even if it does, there will be some logs, but there's no logs for an hour except the while loop's log. Also can't hit the service's endpoints anymore, that also supports what I'm saying, it has terminated everything by the time it got to StopAsync method

I had a look at the implementation of IServer.StopAsync in IISHttpServer here.

It just stops processing incoming request, that's it. Whatever it is that keeps the scheduler running it's not related to this. Without seeing any of your code it's really hard to tell what the issue might be. Are you able to produce a small sample that can replicate the issue?

Yes, would be very helpful to see the code for the invocable / task that's causing issues :)

@EtkaBo do you still need help on this?