terminal (cmd or powershell) only work for a few seconds - latest 1.0.60 release

[schories]

Describe the bug Only for the first 5 to 10 seconds either classic cmd or powershell - no matter whether admin or user based - work flawlessly. After that no new output is made, when commands are run. Finally, after another 10 to 30 seconds later the terminal session is closed automatically, without any error message or any prior warning. All Windows 10 LTSB/LTSC 1607, latest patchlevel, clients are affected. All browsers, latest Edge or latest Chrome.

To Reproduce Steps to reproduce the behavior:

1. Pick any Windows 10 LTSB/LTSC client.
2. Connect/open a terminal session.
3. Type "dir"
4. Type "dir" again.
5. The output of the 2nd "dir" command will be incomplete - please see attached screenshots.
6. After typing more commands no new output is being shown.
7. Finally the terminal session will be closed by the system, not by choice of the user.

Expected behavior Terminal - either classic cmd or powershell, no matter whether admin or user based - should continue to work, means showing output of commands.

Screenshots

Server Software:

• OS: Windows Server 2022
• Virtualization: VMware
• Network: "WAN only" mode
• Version: 1.0.60
• Browser: Edge 103.0.1264.71 (Official Build) (64-Bit), Chrome 103.0.5060.134 (Official Build) (64-Bit)

Remote Device:

• Device: VM
• OS: Windows 10 LTSC 1607
• Version: 1607
• Current Core Version (if known): "Microsoft Windows 10 Enterprise 2016 LTSB - 14393"

Your config.json file

{
   "settings": {
      "cert": "xxxxx.yyyyyy.domain", 
      "_minify": true, 
      "wanonly": true, 
      "selfupdate": true, 
      "minify": true,
      "allowframing": true,
      "allowLoginToken": true
   }, 
   "domains": {
     
   }
}

Please feel free to add information on how to support debugging this issue, so that I can supply more debug data.

[schories]

Update 1:

• A fresh install of a "vanilla" Windows 10 21H2/19044 64 Bit - is not affected. This is good news.

• Therefore I'll now check whether a freshly installed "vanilla" Windows 10 LTSB/LTSC 1607/14393 64 Bit is also affected or not. I want to find out, whether this issue is specific to a certain Windows 10 version (e.g. LTSB/LTSC) - or caused/triggered by some of our settings/configuration.

Thank you once again!

[krayon007]

On your windows system that doesn't work, can you type osinfo in the console tab, and let me know what it says for ConPTY support?

[schories]

Of course,

• on the affected systems: OS: Microsoft Windows 10 Enterprise 2016 LTSB - 14393 [ConPTY: NO]
• on the non affected system: OS: Microsoft Windows 10 Pro for Workstations - 21H2/19044 [ConPTY: YES]

As ConPTY was spoken about first in 2018 - https://devblogs.microsoft.com/commandline/windows-command-line-introducing-the-windows-pseudo-console-conpty/ - and introduced with Windows 19H1. I haven't found information whether ConPTY can be added to LTSB (1607). At least all following releases should include it: https://docs.microsoft.com/en-us/windows/whats-new/ltsc/

Any chance to fix this issue without ConPTY? Please let me know how can help debugging this, too (as I am new to MC2). Thank you! :-)

[schories]

Update 2:

• A fresh install of a "vanilla" Microsoft Windows 10 Enterprise 2016 LTSB - 14393 64 Bit - is affected, too.
• This means the issue is not caused/triggered by some of our settings/configuration.
• It is happening when ConPTY is not present, [ConPTY: NO].

A quick look into this repo makes me think,

• that /agents/modules_meshcore/win-terminal.js is the code for the classic terminal [ConPTY: NO],
• while /agents/modules_meshcore/win-virtual-terminal.js is the ConPTY implementation [ConPTY: YES]

I will now check,

• whether it is just the output on the terminal screen that is missing - or the commands are not run at all.
• try to understand /agents/modules_meshcore/win-terminal.js :-)

Please kindly let me know how I can provide better feedback and any other useful insights.

Thank you!

[krayon007]

Interesting. I have a system that lacks ConPTY, I'll see if it behaves the same way. I know I used it a few weeks ago, I'll see if it still works.

[schories]

So the results are at least consistent. Maybe MSFT introduced this issue with a recent update.

Will hopefully have some more time today to dig deeper into the details/code.

Thank you very much!

[JordanGitHub1]

Hi Guys,

We are experiencing the same kind of behavior schories described, only this time not on endpoints but on servers. We first thought it had something to do with the antivirus but after quite a bit of testing we got different results: • Microsoft Windows Server 2019 Datacenter - 17763; No issues related to powershell in mesh. It works fine. • Microsoft Windows Server 2016 Datacenter - 14393; Same issues as schories described. • Microsoft Windows Server 2012 R2 Datacenter - 9600; Same issues as schories described. • Microsoft Windows Server 2008 R2 Standard - 7601; Same issues as schories described.

We have tried to reinstall the Mesh agent on those machines but the powershell issue remains. It looks like it is an OS based issue here.

[schories]

Hi,

@MartinMa and I have condensed our results into the following discussion:

https://github.com/Ylianst/MeshCentral/discussions/4898

@krayon007 @Ylianst Your kind advice is highly appreciated as we want to contribute a decent fix but are currently stuck. :-/

Thank you once again

Alexander

[MartinMa]

I made a little test application that starts a console in a new process and logs all console events to narrow down this issue independently of MeshCentral/MeshAgent.

The source can be found at WinTerminalTest in case you want to try it yourself.

The test results are quite interesting.

On my development machine everything works as expected. This is an excerpt from the log. Windows Version 20H2 (Build 19042.2364)

EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 21, Right: 0, Bottom: 21]
EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 22, Right: 0, Bottom: 22]
EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 24, Right: 0, Bottom: 24]
EVENT_CONSOLE_CARET: [X: 49, Y: 24, Flags: CONSOLE_CARET_VISIBLE]
EVENT_CONSOLE_UPDATE_SIMPLE: [X: 49, X: 24, Char: 100, Attr: 7]
EVENT_CONSOLE_CARET: [X: 50, Y: 24, Flags: CONSOLE_CARET_VISIBLE]
EVENT_CONSOLE_UPDATE_SIMPLE: [X: 50, X: 24, Char: 105, Attr: 7]
EVENT_CONSOLE_UPDATE_SIMPLE: [X: 51, X: 24, Char: 114, Attr: 7]
EVENT_CONSOLE_CARET: [X: 52, Y: 24, Flags: CONSOLE_CARET_VISIBLE]
EVENT_CONSOLE_UPDATE_SCROLL: [dx: 0, dy: -1]
EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 24, Right: 0, Bottom: 24]
EVENT_CONSOLE_UPDATE_SCROLL: [dx: 0, dy: -1]
EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 24, Right: 0, Bottom: 24]
EVENT_CONSOLE_UPDATE_SCROLL: [dx: 0, dy: -1]
EVENT_CONSOLE_UPDATE_SCROLL: [dx: 0, dy: -1]
EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 24, Right: 0, Bottom: 24]
EVENT_CONSOLE_UPDATE_SCROLL: [dx: 0, dy: -1]
EVENT_CONSOLE_UPDATE_SCROLL: [dx: 0, dy: -1]
EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 24, Right: 0, Bottom: 24]
EVENT_CONSOLE_UPDATE_SCROLL: [dx: 0, dy: -1]
EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 24, Right: 0, Bottom: 24]
EVENT_CONSOLE_UPDATE_SCROLL: [dx: 0, dy: -1]
EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 24, Right: 0, Bottom: 24]

The update simple event is actually me typing the dir command. As you can see, as soon as the console reaches the end of the screen buffer (Bottom: 24), there is a scroll event happening that shifts the contents of the screen buffer up by one row. Thus, making room to draw a new line to the last row. By the way, the two consecutive scroll events result in an empty row on the console.

The following log excerpt has been created on an affected system. Windows Version 1607 (Build 14393.5582)

EVENT_CONSOLE_CARET: [X: 26, Y: 15, Flags: CONSOLE_CARET_VISIBLE]
EVENT_CONSOLE_UPDATE_SIMPLE: [X: 26, X: 15, Char: 100, Attr: 14]
EVENT_CONSOLE_UPDATE_REGION: [Left: 27, Top: 15, Right: 27, Bottom: 15]
EVENT_CONSOLE_CARET: [X: 27, Y: 15, Flags: CONSOLE_CARET_VISIBLE]
EVENT_CONSOLE_UPDATE_REGION: [Left: 26, Top: 15, Right: 26, Bottom: 15]
EVENT_CONSOLE_UPDATE_SIMPLE: [X: 28, X: 15, Char: 114, Attr: 8]
EVENT_CONSOLE_UPDATE_REGION: [Left: 26, Top: 15, Right: 26, Bottom: 15]
EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 17, Right: 0, Bottom: 17]
EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 19, Right: 0, Bottom: 19]
EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 20, Right: 0, Bottom: 20]
EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 21, Right: 0, Bottom: 21]
EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 22, Right: 0, Bottom: 22]
EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 23, Right: 0, Bottom: 23]
EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 24, Right: 0, Bottom: 24]
EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 24, Right: 0, Bottom: 24]
EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 24, Right: 0, Bottom: 24]
EVENT_CONSOLE_UPDATE_REGION: [Left: 50, Top: 24, Right: 50, Bottom: 24]
EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 24, Right: 0, Bottom: 24]
EVENT_CONSOLE_UPDATE_REGION: [Left: 50, Top: 24, Right: 50, Bottom: 24]
EVENT_CONSOLE_UPDATE_REGION: [Left: 0, Top: 24, Right: 0, Bottom: 24]

As you can see, the event EVENT_CONSOLE_UPDATE_SCROLL is missing completely. When the end of the screen buffer is reached, we only receive EVENT_CONSOLE_UPDATE_REGION events that tell us, that the last row was updated.

That way we are unable to detect when the contents of the console screen buffer have been shifted up. It actually looks like a Windows bug. But I couldn't find anything about it. I wonder if it worked before (say in Windows 8.1 or Windows 7).

Solution proposal What do you think about using winpty as fallback mechanism when ConPTY is not available on the system. This software package is used by VSCode to display the integrated terminal on older systems. It polls the screen buffer every 25ms for changes, scrapes its content with some complicated logic and generates a corresponding stream of output.