WebJobs Shutdown signal not working

[mathewc]

WebJobsShutdownWatcher is designed to monitor a sentinel file that the Kudu continuous webjobs infrastructure uses to signal that the WebJob is shutting down. In v2 this shutdown signal would cause RunAndBlock to complete, code here stopping the WebJob exe and allowing any cleanup code to run.

In v3, we're not honoring this - causing a shutdown signal that is picked up by the WebJobsShutdownWatcher does not cause the await host.RunAsync() to complete, allowing any subsequent cleanup code to run. While we do handle the shutdown file notification and trigger token cancellation, it appears that currently this shutdown doesn't cause IHostedService.StopAsync to be called, meaning any hosted service registered by the SDK or users are not stopped. Likely WebJobsShutdownWatcher needs to use IHostLifetime to call StopAsync when triggered.

We shouldn't be ignoring the shutdown signal - it's part of graceful shutdown. As it is now the WebJob will continue running until Kudu kills the process.

[divinci]

At last some information regarding why my shutdown code is not working :)

Thanks @mathewc do you have information for a workaround in the mean time?

[divinci]

This doesn't seem to work.

var host = builder.Build();
var cancellationToken = new WebJobsShutdownWatcher().Token;
cancellationToken.Register(async () => await host.StopAsync());

using (host)
{
    await host.RunAsync(cancellationToken);
}

[divinci]

This doesn't seem to work.

var host = builder.Build();
var cancellationToken = new WebJobsShutdownWatcher().Token;
cancellationToken.Register(async () => await host.StopAsync());

using (host)
{
    await host.RunAsync(cancellationToken);
}

After some debugging, this code allows for graceful shutdown in my local environment when I introduce the environment variable WEBJOBS_SHUTDOWN_FILE and manually create the file. But when deployed to Azure, something means it will not work :/

[divinci]

Any news on this issue @fabiocav @mathewc?

[aherrick]

Just to be clear, there isn't any workaround to stopping a continuous web job via the azure portal? (not Kudu process kill)

[divinci]

Just to be clear, there isn't any workaround to stopping a continuous web job via the azure portal? (not Kudu process kill)

I can't find one.

[alhardy]

Having the same issue

[JSCProjects]

Facing same issue here.

Other question related to webjobs how do you deploy your webjobs? I'm also facing deployment issues with v3

[joshidp]

Any update, please?

Thanks

[BobbyCannon]

I also cannot seem to find the correct answer on how to gracefully shutdown a continuous web job.

[Socolin]

@BobbyCannon I'm using something like this as a workaround

_host = builder.Build();
var webJobsShutdownWatcher = new WebJobsShutdownWatcher();
webJobsShutdownWatcher.Token.Register(async () => { await _host.StopAsync(); });
await _host.StartAsync();
var applicationLifetime = _host.Services.GetService<IHostApplicationLifetime>();
var jobHost = _host.Services.GetService<IJobHost>();
var task = jobHost.CallAsync(nameof(Functions.RunAsync), cancellationToken: applicationLifetime.ApplicationStopping);
await _host.WaitForShutdownAsync();
await task;

[JSCProjects]

The problem for me was that I deployed the webjob with azure devops with deployment method "Auto-detect" after changing it to "Zip Deploy" I was able to stop the webjob normally

[BobbyCannon]

Where is this setting? We deploy our jobs with our website deployment...

[ibradwan]

Any updates? Or is there any consistent workaround to trigger some shutdown logic before the host kills the process?

[mathewc]

Fix is checked in (see https://github.com/Azure/azure-webjobs-sdk/commit/01786831f084c5e7a5afb4301464e2d91bfe17af). Will be in the next package release.

[DRAirey1]

I'm using Microsoft.Azure.WebJobs 3.0.31. Not ever getting a signal the WebJob has been shut down with this code (I'm shutting down from Azure Portal->App Service->WebJobs page:

using Microsoft.Azure.WebJobs;

namespace WorkerService
{
    public class WorkerService : IHostedService
    {
        private readonly ILogger logger;

        private readonly IHostApplicationLifetime appLifetime;

        private readonly WebJobsShutdownWatcher webJobsShutdownWatcher = new ();

        private Task? task;

        /// <inheritdoc/>
        public WorkerService(ILogger<WorkerService> logger, IHostApplicationLifetime appLifetime)
        {
            this.logger = logger;
            this.appLifetime = appLifetime;
        }

        public Task StartAsync(CancellationToken cancellationToken)
        {
            this.logger.LogInformation("called StartAsync");
            this.appLifetime.ApplicationStarted.Register(OnStarted);
            this.appLifetime.ApplicationStopping.Register(OnStopping);
            this.appLifetime.ApplicationStopped.Register(OnStopped);

            this.task = this.DoWorkAsync(this.webJobsShutdownWatcher.Token);
            return this.task;
        }

        public Task StopAsync(CancellationToken cancellationToken)
        {
            this.logger.LogInformation("called StopAsync");
            return Task.CompletedTask;
        }

        private async Task DoWorkAsync(CancellationToken cancellationToken)
        {
            this.logger.LogInformation("Starting DoWorkAsync");

            int counter = 0;
            while (!cancellationToken.IsCancellationRequested)
            {
                if (this.appLifetime.ApplicationStopping.IsCancellationRequested)
                {
                    this.logger.LogInformation("App Service shutdown requested.");
                }

                if (this.webJobsShutdownWatcher.Token.IsCancellationRequested)
                {
                    this.logger.LogInformation("Web Job shutdown requested.");
                }

                if (counter++ > 10)
                {
                    this.logger.LogInformation("Forcing application shutdown.");
                    this.appLifetime.StopApplication();
                }

                this.logger.LogInformation($"Phase V: Doing work at {DateTime.Now}");
                await Task.Delay(10000, this.appLifetime.ApplicationStopping);
            }

            this.logger.LogInformation("Exiting DoWorkAsync");
        }

        private void OnStarted()
        {
            this.logger.LogInformation("OnStarted has been called.");
        }

        private void OnStopping()
        {
            this.logger.LogInformation("OnStopping has been called.");
        }

        private void OnStopped()
        {
            this.logger.LogInformation("OnStopped has been called.");
        }
    }
}