Invoke/InvokeAsync hangs while debugging until VS restart

[sherman89]

I noticed while testing my console application that at some point after starting/stopping debugging, the InvokeAsync and Invoke methods hanged and never returned, this kept happening after that time all the time until I finally restarted Visual Studio then the problem disappeared.

Unfortunately I don't have any steps to reproduce this issue, don't have any stack traces, and it hasn't happened in a while anymore, but thought it might still be a good idea to open an issue about it.

The call: return await rootCommand.InvokeAsync(args);

Thank you for this library! Can't wait for a documented and stable version :)

EDIT: If I recall correctly, the version of VS I was using at the time was 16.4.x (sorry don't remember, should have opened an issue right there and then!)

[jonsequitur]

This sounds like it could be an async deadlock. Did you happen to have any code that called Result or Wait on a Task?

[sherman89]

@jonsequitur

No I didn't have any calls to Result anywhere, everything was awaited correctly. When I encountered the issue, I replaced InvokeAsync with Invoke, but still got the same result strangely.

[derekchristensen]

I'm not sure if my issue is related but it sounds similar. I'm experiencing a deadlock pretty regularly when I use Ctrl+C to exit my application. The CancellationToken cancels correctly and my command handler (running on a worker thread) exits with a return 0. After my handler exits, the app deadlocks. It looks to me like the call to remove the Console.CancelKeyPress event handler is waiting for a lock while the ManualResetEventSlim that is created as part of CancelOnProcessTermination is also being waited on by my main thread. My app is not using Result or Wait either, just normal await calls from an async Main.

Here are a couple stack traces from my dnSpy debugging:

[jonsequitur]

@derekchristensen What version are you using and can you provide a repro?

[derekchristensen]

I started with [email protected] but I've upgraded to [email protected] in my sample below and the problem persists. I'm also targeting net472. I haven't tested against netcoreappXX.

I found that I had to load an assembly with Assembly.LoadFile to cause the lock to happen.

Here's repro https://github.com/derekchristensen/DeadlockSample

[antineutrino]

Same problem here as @derekchristensen. I'm also loading assemblies with Assembly.Load in the AppDomain.CurrentDomain.AssemblyResolve event handler.

[menees]

Same problem here as @derekchristensen. But thanks to his Jun 4, 2021 mention of CancelOnProcessTermination , I was able to workaround the problem by removing the call to CancelOnProcessTermination. Now, I manage the cancel key press on my own along with my own CancellationTokenSource, but at least I'm not deadlocking anymore.

To anyone else arriving here, CancelOnProcessTermination is used by CommandLineBuilderExtensions's UseDefaults() method. So, you're using it implicitly unless you're creating your own CommandLineBuilder and initializing it explicitly without the CancelOnProcessTermination call and without a UseDefaults call.

[pharring]

The problem is that on .NET Framework (Windows only):

1. The Ctrl+C signal from the OS arrives on one thread with the "ConsoleState" critical section held.
2. The Framework's "Break" handler queues work to the thread pool thread to call the handlers and blocks, waiting for them all to complete (so it can return the "Cancel" value back to the OS)
3. ConsoleCancelEventHandlers are called from the thread pool thread. CommandLine's CancelOnProcessTermination implementation cancels the CancellationToken.
4. Any registered handlers on the CancellationToken run. Typically any blocking tasks detect this and throw a TaskCanceledException. Any awaiters, will re-throw that exception, all the way out to the middleware set up in CancelOnProcessTermination. All of this happens synchronously on the same threadpool thread. And, all the time, the OS is blocked waiting for the queued work item to complete with the ConsoleState critical section still held.
5. The finally block of that middleware runs and tries to remove the CancelKeyPressEvent handler. This becomes a P/Invoke to KernelBase.dll!SetConsoleCtrlHandler which tries to enter the ConsoleState critical section, resulting in deadlock.

Note: .NET Core's Ctrl+C handling is significantly different, so it doesn't suffer from this deadlock.

The workaround here is to throw in a await Task.Yield(); on the cancellation code path. That can be done in middleware:

CommandLineBuilder commandLineBuilder = new CommandLineBuilder(rootCommand)
    .CancelOnProcessTermination()
#if NETFRAMEWORK
    .AddMiddleware(async (context, next) =>
    {
        try
        {
            await next(context);
        }
        catch (OperationCanceledException)
        {
            // Avoid deadlock when CancelOnProcessTermination's middleware
            // tries to unhook the Ctrl+C handler.
            await Task.Yield();
            throw;
        }
    })
#endif
    .UseVersionOption()
    .UseHelp();

The order is significant. This middleware must be added after CancelOnProcessTermination. It can also be used with UseDefaults (which includes CancelOnProcessTermination)