conpty process blocked update on Windows

[connor4312]

I attempted to update Insiders today using the usual "apply update and restart" action and ran into the following error. It looks like some console-related processes blocked update. No VS Code windows were open.

The command line of the process doing the blocking was:

"c:\Users\conno\AppData\Local\Programs\Microsoft VS Code Insiders\resources\app\node_modules.asar.unpacked\node-pty\build\Release\conpty\OpenConsole.exe" --headless --inheritcursor --width 80 --height 30 --signal 0x638 --server 0x69c

I am using the new windowsUseConptyDll setting, if that matters.

[joaomoreno]

https://github.com/microsoft/vscode/issues/225160

[deepak1556]

Upon killing the processes hold the lock and taking the update, the restored terminals show as disconnected, and I could not create new terminals until I restarted vscode:

Edit: so far 4 for 4 on hitting this each time I update Insiders

Originally posted by @connor4312 in https://github.com/microsoft/vscode/issues/225160#issuecomment-2293757400

[deepak1556]

@connor4312 I am looking to repro but maybe I am missing some steps, can you confirm. Is there any particular set of terminal commands that trigger this issue ?

• Launch Inisders with windowsUseConptyDll enabled
• Open terminals with long running tasks, ex: yarn in vscode repo
• Check and restart for update (tested with both backgroundupdates enabled and disabled)
• Update completed without issues

[connor4312]

That's right. But specifically, I usually have the vscode-copilot repo open which has its long running watch tasks, so that may also be a part of it.

[deepak1556]

Didn't have any luck with the copilot repo as well, can you try the following steps before applying the next update

• Open Windows Performance Recorder from system start menu
• Enable the following settings, in the resource analysis section specifically CPU usage, Disk I/O activity, File I/O activity, Registry I/O activity, Handle usage

• Start the recorder
• Perform the update
• Once the access denied error happens, stop the recorder and save the profile
• Send the profile to [email protected] instead of attaching to this issue, since it will contain sensitive information.

[connor4312]

Thanks, sent!

[deepak1556]

Thanks for the profile, based on the process lifetimes I was able to backtrack the steps that might repro the issue and I now have a reliable repro,

• Launch insiders with windowsUseConptyDll on the vscode-copilot repo
• Background OpenConsole.exe process will be created for each of the npm tasks
• Use the debug task to launch Copilot extension
• Close the debugger window
• the debugee window will close shortly but OpenConsole.exe process is leaked and end up locking the resources folder

I feel the scenario during the update might have been the same, @connor4312 just to validate this scenario can you check if update fails when you don't have any debugging session.

[connor4312]

I updated Insiders today without a debugging session and didn't run into the issue 🙂

[Tyriar]

I saw this just now:

[connor4312]

This broke my Insiders build today, I think I tried to cancel setup when the issue happened and then I opened Insiders again to see if that would work without digging out LockHunter. The update failed and then gave the "antivirus might have interfered with the update" error, and I had to reinstall Insiders at that point.

[connor4312]

Today and for the mid-day Insiders yesterday I was much more careful about the update

just to validate this scenario can you check if update fails when you don't have any debugging session.

This does happen even when there is no debugging session active. In both cases I had the task terminals open for Copilot Chat, so I suspect maybe task reconnection logic is related to this issue

[Tyriar]

@deepak1556 was going to check this out when he's free 🙏

[deepak1556]

Didn't have much cycles this milestone due to my vacation, I will look into this in the next debt week.

[sbatten]

By the way I've been seeing this every day since I have been debugging copilot chat simulations

[deepak1556]

Starting with today's insider this issue should be fixed, please selfhost again with windowsUseConptyDll and let me know of issues that you face.

[connor4312]

Unfortunately I got this updating Friday -> Today's Insiders

[deepak1556]

@connor4312 the openconsole.exe process should have an associated client process running that would be keeping it from exiting, like node.js process or something (these are process that you spawn from the integrated terminal). Can you check the process tree to see what process are associated with the running openconsole process, it would give a hint on how to repro the issue on my side. Thanks!

[connor4312]

Will do next time I hit this

[connor4312]

I don't see much of a tree in Process Exporer, but here's some info:

I also took a minidump of the process if that would be helpful

[deepak1556]

Thanks for checking, yes can you share the dump file to [email protected]

Looks like having the job handle https://github.com/microsoft/node-pty/pull/755 would be the right fix given we force close application with background updates and the pty agent process would be unable to drain data from the openconsole process in this case. I actually reverted that change for a more graceful shutdown on the pty side https://github.com/microsoft/node-pty/pull/759/commits/667e93f33d68a7be25f1e550cba6ce723961e400. If the minidump shows that to be the case, I will go ahead and bring back this fix.

[deepak1556]

Some more updates from the discussion

In any case, the ConsoleIoThread is stuck in an IO call on the server handle. Due to the lack of backtrace it's unclear what call it is, but it's very likely to be a read call. This would indeed mean that either the node-pty client is still holding the PTY handle or a client is still connected. A full memory dump or similar would help shed more light on this.

Learnt that we can inspect the following variable

::Microsoft::Console::Interactivity::ServiceLocator::s_globals.ciConsoleInformation.ProcessHandleList

To identify the clients that are holding onto the openconsole process.

@connor4312 would you be able to collect a full process dump, I would like to find the root cause of this issue before pushing any workarounds.

To get a full memory dump, you can use https://learn.microsoft.com/en-us/sysinternals/downloads/procdump

procdump -ma <PID-of-leaking-openconsole>