Input on line at end of scrolling region (set with DECSTBM) causes cursor to move down outside the scroll region

[DavidSpickett]

Windows Terminal version

1.21.11141.0

Windows build number

10.0.22631.5335

Other Software

This is a standalone program that replicates what LLDB is seeing (https://github.com/llvm/llvm-project/issues/134846) when enabling LLDB's status line feature. This statusline feature is only present in development (main branch) builds of llvm 21 and is currently disabled on Windows due to this bug.

I'm attaching a test program that does the same thing without referring to any lldb code. I can provide a patch to re-enable it in LLDB if anyone wants it, but probably easier to ask me to do it if you want to know something about LLDB specifically.

console_test.zip

There is C program that creates a basic REPL with a persistent status bar. You can type "hello", nothing, or random letters then press enter to get it to respond with some output.

The makefile uses cl.exe to build, I can get the versions for that if it would be useful to you. I should note that this is Windows on Arm, I have not been able to try a Windows x64 machine.

Steps to reproduce

Open a Visual Studio developer command prompt and build the application in there.

cl.exe test.c

(makefile is for Linux/gcc)

Then run it in a Windows Terminal tab. Normal command prompt or PowerShell, both show this problem.

.\test.exe

You should see this at the top of the terminal:

Simple REPL. Type commands below:
> <-- your cursor is here, one char beyond the '>'

And at the bottom of the screen:

Status (draw 1): Type 'hello' to greet. Ctrl+C or "q" to exit.

(though the ctrl-c part isn't implemented for Windows, it's not relevant to this problem)

draw 1 means it's the first time it's drawn the bar, enter redraw to redraw it, if you want to check for overlapping text issues. In theory this program never needs to redraw the status bar because it's outside of the scroll area.

Enter some input and press enter:

Simple REPL. Type commands below:
> hello
hello!
> <-- cursor is here again

It acts as any other REPL, going down the screen towards the status bar as more output is generated.

Expected Behavior

That you can continue entering input and pressing enter, and the output will continue down the screen until the status bar. When you are right above the status bar, input is echoed on the last line of the scroll window and new output lines cause all the content to scroll up one row, so as not to overwrite the status bar.

This is what happens on Linux using Ubuntu's default terminal.

Empty command
> <-- your cursor is here
Status (draw 1): Type 'hello' to greet. Ctrl+C or "q" to exit.

Input echoes to the current line:

Empty command
> h<-- your cursor is here
Status (draw 1): Type 'hello' to greet. Ctrl+C or "q" to exit.

New output lines cause content to scroll up:

Empty command
> h
Unknown command: h
> <-- your cursor is here 
Status (draw 1): Type 'hello' to greet. Ctrl+C or "q" to exit.

Rather than overwrite the status line.

The terminal should act this way no matter how many times the user does this. It should keep scrolling the content up.

Actual Behavior

The problem is when we get down to just above the status bar. This is the first time you reach that position:

Empty command
> <-- your cursor is here
Status (draw 1): Type 'hello' to greet. Ctrl+C or "q" to exit.

Type something and press enter:

Empty command
> abc
Unknown command: abc
>
Status (draw 1): Type 'hello' to greet. Ctrl+C or "q" to exit.

So far, so good. Now try that again, here I typed "z":

Empty command
> abc
Unknown command: abc
>
Stztus (draw 1): Type 'hello' to greet. Ctrl+C or "q" to exit.
   ^
    \----- the cursor moved vertically down one row before printing the "z" I entered.

It has now started to write over the status bar.

Here it is in image form.

When we first reach the bottom of the scroll window:

Then enter a command:

That works, but the second time, it does not work:

From here, the program is stuck in some sort of "one line mode" where it will only output and print on that last line. I guess because it knows it is outside the scroll window, so it can't scroll up into that.

It manages to work once, which I find suspicious.

[DavidSpickett]

Two things I know might be incorrect but I don't think change the behaviour here:

• We use column 0 and row 0. Columns and rows start from 1, but using 0 seems to work the same way on Linux and Windows.
• We get the terminal height from dwMaximumWindowSize. I have seen some sources say to use srWindow, but in my testing they returned the same dimensions.

[DavidSpickett]

https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences#scrolling-margins is the escape code we rely on for this.

[lhecker]

We get the terminal height from dwMaximumWindowSize. I have seen some sources say to use srWindow, but in my testing they returned the same dimensions.

FWIW the difference is noticeable when you use the older console host (conhost). The relevant code is here: https://github.com/microsoft/terminal/blob/4abc041eb78902708e1844902e40a88d0bc3d50a/src/host/screenInfo.cpp#L404-L426

!...IsHeadless() in this case means "if this is not a PTY", which the older console host indeed isn't.

[DavidSpickett]

FWIW the difference is noticeable when you use the older console host (conhost).

Thanks! I do see a difference in conhost. I will check what LLDB should be using for its purposes.

[DHowett]

@lhecker Interesting... I see consecutive cursor positioning sequences--one for every even numbered line (there is one line between prompts, so that makes sense) up until it emits one for the final odd-numbered line--coming out of what I am guessing is COOKED_READ. This may be a case we weren't prepared for: mixing non-VT read with VT-write-mediated terminal features like scrolling regions.

␛[2;3H␍␊
Empty␣command␍␊
␍>␣␛[13;28;13;0;0;1_␛[13;28;13;1;0;1_␛[4;3H␍␊
Empty␣command␍␊
␍>␣␛[13;28;13;0;0;1_␛[13;28;13;1;0;1_␛[6;3H␍␊
Empty␣command␍␊
␍>␣␛[13;28;13;0;0;1_␛[13;28;13;1;0;1_␛[8;3H␍␊
Empty␣command␍␊
␍>␣␛[13;28;13;0;0;1_␛[13;28;13;1;0;1_␛[10;3H␍␊
Empty␣command␍␊
␍>␣␛[13;28;13;0;0;1_␛[13;28;13;1;0;1_␛[12;3H␍␊
Empty␣command␍␊
␍>␣␛[13;28;13;0;0;1_␛[13;28;13;1;0;1_␛[14;3H␍␊
Empty␣command␍␊
␍>␣␛[13;28;13;0;0;1_␛[13;28;13;1;0;1_␛[16;3H␍␊
Empty␣command␍␊
␍>␣␛[13;28;13;0;0;1_␛[13;28;13;1;0;1_␛[18;3H␍␊
Empty␣command␍␊
␍>␣␛[13;28;13;0;0;1_␛[13;28;13;1;0;1_␛[20;3H␍␊
Empty␣command␍␊
␍>␣␛[13;28;13;0;0;1_␛[13;28;13;1;0;1_␛[22;3H␍␊
Empty␣command␍␊
␍>␣␛[13;28;13;0;0;1_␛[13;28;13;1;0;1_␛[24;3H␍␊
Empty␣command␍␊
␍>␣␛[13;28;13;0;0;1_␛[13;28;13;1;0;1_␛[26;3H␍␊
Empty␣command␍␊
␍>␣␛[13;28;13;0;0;1_␛[13;28;13;1;0;1_␛[28;3H␍␊
Empty␣command␍␊
␍>␣␛[13;28;13;0;0;1_␛[13;28;13;1;0;1_␛[29;3H␍␊

Note the lines in CSI H: 4, 6, 8, 10 ... 26, 28, 29. Which is expected, considering that we pushed a single line off the top of the screen and needed to put the prompt in the second from bottom line. However: we follow that up by positioning the next one at 30. Once we're trapped under the scroll margin, we're done for.

[DHowett]

Now, this doesn't work in conhost at all. We just write off the bottom of the screen and push the statusbar up (even with the fix to use srWindow and the fix to enable ENABLE_VIRTUAL_TERMINAL_PROCESSING) :)

[lhecker]

Oh, it uses the regular cooked read feature in this context? 😢 I suspect this will break even worse when you use our history listing (F7). But that explains things. I wonder how we should fix this...

(Ideally, LLDB would implement its own readline handling to better integrate with its features. It would allow for tab-based autocompletion in the future, etc. If you're familiar with this, it's actually a lot easier to implement your own readline than you may think.)

[DHowett]

Thanks for the comprehensive report, and the good repro case!

Right now, it exists in the unexplored space between "using Windows APIs" (fgets in this case being a wrapper around ReadConsoleInput) and "using VT". It's not something we're likely to prioritize fixing in the near-term, but we'll keep it on the backlog.

As an example, Python recently moved to their own line editor for their REPL, and it unlocked a lot of potential around how they display input (with syntax highlighting) and gave them the ability to support common line editing keys that their userbase may be familiar with.

I wonder if it's in LLDB's best interest to do the same?

Even on lldb's other platforms, it may afford your users an improved experience!

[j4james]

It's worth noting that this issue can be triggered with just the cmd shell. For example if you echo Ctrl+[ then [1;10r, you'll have created a little window in the top half of the screen. And you can perform a directory listing there which will be constrained within the margins, but once you start pressing Enter at the prompt, you'll end up breaking out of that window.

I'm almost positive this used to work - it definitely still works in my Windows 10 conhost - so it's kind of a regression.

[DavidSpickett]

Right now, it exists in the unexplored space between "using Windows APIs" (fgets in this case being a wrapper around ReadConsoleInput) and "using VT". It's not something we're likely to prioritize fixing in the near-term, but we'll keep it on the backlog.

Thanks for the assessment. I see we're mixing APIs which is always going to be risky, so the prioritisation makes sense.

(fgets in this case being a wrapper around ReadConsoleInput)

So if we wanted to be all Virtual Terminal (that's what VT means, right?), what would we use here instead? Or to do that does LLDB need to be managing the terminal at a lower level?

Gathering the recommendations from the thread, I think LLDB should:

• Use srWindow to get terminal dimensions, so we work with conhost as well.
• Set ENABLE_VIRTUAL_TERMINAL_PROCESSING for the same reason.
• Either use Windows APIs, or use VT (by which you mean terminal escape codes and the like?), but not both. For instance, we could use scrollconsolescreenbuffer instead of the ASNII escape code. Though this page says "Our preferred modern solution focuses on virtual terminal sequences for maximum compatibility in cross-platform scenarios." so perhaps we should go all in on VT instead.
• Consider writing our own editline, which I think would allow us to "manually" get this status bar effect, and the ability to build other features with it. Editline for Windows has been a gap for a long time, on Linux we use libedit and some people have tried building Windows lldb with cross-platform ports of it but it has never worked properly.

[lhecker]

I'm almost positive this used to work - it definitely still works in my Windows 10 conhost - so it's kind of a regression.

Oh. I wonder what I did to break this... In fact, how did this work in Windows 10? Did the old cooked read not circumvent the VT stack and its margins entirely? 🤔

I see we're mixing APIs which is always going to be risky, so the prioritisation makes sense.

Yeah, the problem is the usual in IT: We wanted to improve X and so we had to "break" Y. In this case X is our VT-correctness (we sometimes call it "passthrough mode"; see: #17510) and Y is this cooked read functionality. Previously, cooked read would use console APIs internally (the thing we've said is not preferred anymore), and we had to change that.

Either use Windows APIs, or use VT (by which you mean terminal escape codes and the like?), but not both.

We try our best to make both APIs interact somewhat well with each other. This includes these "cooked reads" here. But it is definitely true, that either only using console APIs or only VT APIs will be more stable.

I strongly recommend only using VT APIs, because if you properly batch/buffer your stdout, VT will be noticeably faster. It also has a lot more features of course and works cross-platform.

Editline for Windows has been a gap for a long time, on Linux we use libedit and some people have tried building Windows lldb with cross-platform ports of it but it has never worked properly.

Oh, I see now why you're using our cooked read implementation. I wonder if we (the terminal team) should write a simple drop-in GNU readline implementation one day, so projects like LLDB can use it. 🤔 But as mentioned above (re: Python REPL), if you have the ability to do so, writing your own editline will definitely allow for a better integration with lldb in the future.

Until then, we should see if we can fix this issue on our side. Hopefully we can. This is the relevant code: https://github.com/microsoft/terminal/blob/ad149228744ca1b2ad1ee50bfbe6eb474b7141a4/src/host/readDataCooked.cpp#L902-L1290

Apropos, relatedly, check out this TUI application: https://github.com/al13n321/nnd

[j4james]

Oh. I wonder what I did to break this... In fact, how did this work in Windows 10? Did the old cooked read not circumvent the VT stack and its margins entirely? 🤔

Now that I think about it, I'm not sure that you did break this. I wouldn't be surprised if it was actually me that broke it, long before you rewrote cooked read. Because the Windows 10 conhost is really old, and if I remember correctly, the original margin implementation was actually managed somewhere in the conhost internals. It probably got moved out when I refactored everything to merge the Windows Terminal VT implementation with the conhost AdaptDispatch implementation.

So assuming the cooked read needs to know the margin boundaries, that may well have been something that was originally accessible in CONSOLE_INFORMATION (or somewhere like that), and now would need to be determined via other means.

Although ideally if the new cooked read is meant to be a pure VT implementation, it shouldn't be accessing internal information like that anyway. If you were expecting terminal apps to produce their own readline implementation, they wouldn't have access to the margin boundaries either (other than querying the terminal with an escape sequence).

[MamiyaOtaru]

I believe this is what I am seeing. If so, there is some minimal Powershell to trigger it below. Works fine in the Win10 conhost. Breaks in Win11 Terminal, OpenConsole.exe, conhost

[char]$ESC               = 0x1b
$VT_SAVECURSOR           = "$ESC7"  # Save cursor and attrib
$VT_RESTORECURSOR        = "$ESC8"  # Restore cursor pos and attribs
$VT_SETWIN_CLEAR         = "$ESC[r" # Clear scrollable window size
$VT_CLEAR_SCREEN         = "$ESC[2J" # Clear screen
$VT_CLEAR_LINE           = "$ESC[2K" # Clear this whole line
$VT_RESET_TERMINAL       = "$ESC[r"

# set scroll area
Write-Host "$ESC[1;25r" -NoNewLine
# move cursor
Write-Host "$ESC[1;1H" -NoNewLine

Get-ChildItem
Write-Host "********"

Read-Host "Press Enter "

Get-ChildItem
Write-Host "********"

Read-Host "Press Enter "

Get-ChildItem
Write-Host "********"

Read-Host "Press Enter "

Get-ChildItem
Write-Host "********"

Read-Host "Press Enter "

Get-ChildItem
Write-Host "********"


Write-Host $VT_RESET_TERMINAL -NoNewLine

# move cursor to after scroll window.
Write-Host "$ESC[26;1H" -NoNewLine
Write-Host $VT_CLEAR_LINE -NoNewLine
Write-Host "All Done!"