Windows-Containers Asp.net Framework 4.7.2 graceful shutdown

Do you have a sample asp.net application using .net framework 4.7.2 that shows how to gracefully shutdown when kubernetes sends a sigterm, for example when calling kubectrl stop or kubectrl rollout restart? Examples such as this graceful shutdown are for .net core.

Thanks

Oct 24 '24 05:10 SRJames

Thank you for creating an Issue. Please note that GitHub is not an official channel for Microsoft support requests. To create an official support request, please open a ticket here. Microsoft and the GitHub Community strive to provide a best effort in answering questions and supporting Issues on GitHub.

Oct 24 '24 05:10 github-actions[bot]

I am assuming IIS hosted inside a container here.

IIS already provides mechanisms to react on the pool lifecycle, please take a look at:

https://stackoverflow.com/questions/32145838/how-to-properly-use-iregisteredobject-to-block-app-domain-shutdown-recycle-for

This is an example handler:

public class HostingEnvironmentRegisteredObject : IRegisteredObject
{
    protected ILogger Logger;

    public HostingEnvironmentRegisteredObject(ILogger logger)
    {
        this.Logger = logger;
        HostingEnvironment.RegisterObject(this);
        this.Logger.Debug("Registered in hosting environment");
    }

    public delegate void HostingEnvironmentShutdownHandler(
        ApplicationShutdownReason shutdownReason,
        bool immediate);

    public event HostingEnvironmentShutdownHandler HostingEnvironmentShutdown;

    public void Stop(bool immediate)
    {
        var reason = HostingEnvironment.ShutdownReason;
        this.Logger.Debug($"Hosting Environment Shutdown with reason {reason} and immediate={immediate}");
        this.HostingEnvironmentShutdown?.Invoke(reason, immediate);
        this.Logger.Debug($"Hosting Environment Shutdown Finished");
        HostingEnvironment.UnregisterObject(this);
    }
}

How does this relate to a containerized application? It depends on the container runtime.

If running in docker desktop the shutdown is a total pain. How the engine handles killing will destroy the container with a 5s timeout by deafult.

You can take a look at this Windows Server image which has several adjustments to deal with graceful shutdown:

https://github.com/david-garcia-garcia/windowscontainers/tree/master/servercore2022

https://github.com/david-garcia-garcia/windowscontainers/blob/master/servercore2022/setup/base/assets/entrypoint/refreshenv/SetShutdownTimeout.ps1

This is the code dealing with changing this shutdown timeout:

$global:ErrorActionPreference = if ($null -ne $Env:SBS_ENTRYPOINTERRORACTION ) { $Env:SBS_ENTRYPOINTERRORACTION } else { 'Stop' }

# Default to 20s
$timeout = SbsGetEnvInt -name "SBS_SHUTDOWNTIMEOUT" -defaultValue 20;

# Check if the timeout value was retrieved and is numeric
$timeoutMilliseconds = [int]$timeout * 1000;

reg add hklm\system\currentcontrolset\services\cexecsvc /v ProcessShutdownTimeoutSeconds /t REG_DWORD /d $timeout /f | Out-Null;
reg add hklm\system\currentcontrolset\control /v WaitToKillServiceTimeout /t REG_SZ /d $timeoutMilliseconds /f | Out-Null;

SbsWriteHost "Container shutdown timeout set to $($timeout)s";

When using that image, you can control it through the environment variable:

SBS_SHUTDOWNTIMEOUT

Even after setting this timeout this does not mean that you are actually waiting for IIS to terminate/finish. You should take care of that in the entrypoint by responding to the signal termination.

This is where it gets tricky and the actual implementation depends on what you are using as an entrypoint.

Take a look at this entrypoint implementation to see how to handle termination signal from a powershell script:

https://github.com/david-garcia-garcia/windowscontainers/blob/master/servercore2022/setup/base/assets/entrypoint/entrypoint.ps1

The relevant code is:

$code = @"
using System;
using System.Runtime.InteropServices;

public class ConsoleCtrlHandler {

    [DllImport("Kernel32")]
    public static extern bool SetConsoleCtrlHandler(HandlerRoutine Handler, bool Add);

    public delegate bool HandlerRoutine(CtrlTypes CtrlType);

    public enum CtrlTypes {
        CTRL_C_EVENT = 0,
        CTRL_BREAK_EVENT,
        CTRL_CLOSE_EVENT,
        CTRL_LOGOFF_EVENT = 5,
        CTRL_SHUTDOWN_EVENT
    }

    private static bool _shutdownRequested = false;
    private static bool _shutdownAllowed = true;

    private static System.Collections.Concurrent.ConcurrentDictionary<string, DateTime> _signals = new System.Collections.Concurrent.ConcurrentDictionary<string, DateTime>();

    public static void SetShutdownAllowed(bool allowed) {
        _shutdownAllowed = allowed;
    }

    public static System.Collections.Concurrent.ConcurrentDictionary<string, DateTime> GetSignals() {
        return _signals;
    }

    public static bool GetShutdownRequested() {
        return _shutdownRequested;
    }

    public static bool ConsoleCtrlCheck(CtrlTypes ctrlType) {
        _signals.TryAdd(Convert.ToString(ctrlType), System.DateTime.UtcNow);
        switch (ctrlType) {
            case CtrlTypes.CTRL_CLOSE_EVENT:
            case CtrlTypes.CTRL_SHUTDOWN_EVENT:
                _shutdownRequested = true;
                System.Diagnostics.Stopwatch stopwatch = System.Diagnostics.Stopwatch.StartNew();
                while (!_shutdownAllowed && stopwatch.Elapsed.TotalSeconds < 600) {
                    System.Threading.Thread.Sleep(1000);
                }
                return true;
            default:
                return true;
        } 
    }
}
"@

# Add the C# type to the current PowerShell session
Add-Type -TypeDefinition $code -ReferencedAssemblies @("System.Runtime.InteropServices", "System.Collections.Concurrent");

# Create a delegate for the handler method
$handler = [ConsoleCtrlHandler+HandlerRoutine]::CreateDelegate([ConsoleCtrlHandler+HandlerRoutine], [ConsoleCtrlHandler], "ConsoleCtrlCheck");

# Register the handler
[ConsoleCtrlHandler]::SetConsoleCtrlHandler($handler, $true) | Out-Null;

You can then loop and wait for the termination signal:

[ConsoleCtrlHandler]::SetShutdownAllowed($false);

while (-not [ConsoleCtrlHandler]::GetShutdownRequested()) {
  Start-Sleep -Milliseconds 1000;
}

# Your shutdown code here

# Finally let the sigterm continue
[ConsoleCtrlHandler]::SetShutdownAllowed($true);

I am not 100% sure if WaitToKillServiceTimeout does the trick and properly waits for IIS to shutdown, if not, you should take care of stopping the service in the entrypoint itself after receiving the termination signal.

But all of this is for Docker Desktop... which you are probably not going to use for running your container workloads. In K8S you have lifecycle hooks:

https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/

which is much more reliable than all those sigterm hacks/workarounds.

The image in https://github.com/david-garcia-garcia/windowscontainers/tree/master/servercore2022 handles both scenarios (docker and k8s) by coordinating shutdown through this powershell script:

https://github.com/david-garcia-garcia/windowscontainers/blob/master/servercore2022/setup/base/assets/entrypoint/shutdown.ps1

The entrypoint uses a flag (a file in written to the filesystem to that purpose) to detect if it needs to call shutdown scripts, of they have been called externally.

I hope this servers a starting point.

Oct 27 '24 06:10 david-garcia-garcia

Thank you for that very comprehensive post. Yes, we are using kubernetes via Aws EKS.

Oct 27 '24 12:10 SRJames

@david-garcia-garcia Thanks for that deep dive and all the helpful links!