microsoft
Feature: Add support for Infinite Scrollback
Summary of the new feature/enhancement
I know at least gnome-terminal supports infinite scrollback as a checkbox option in profile settings; it would be great to have that here (I'd love to be able to replace X server+gnome-terminal for WSL).
I checked to make sure that there wasn't an issue for this already, and I saw that you have a TODO comment in the code for this, so I figured I'd open a GitHub issue so that you can use the 👍 to figure out how to rank it.
Proposed technical implementation details (optional)
Looks like the plan is for users to set HistorySize to -1
https://github.com/microsoft/terminal/blob/900d0c3cce39fa191e16b0224fa32c3441f8de24/src/cascadia/TerminalCore/Terminal.cpp#L88
In GNOME Terminal (VTE) we store the scrollback in a compressed and encrypted file. Storing it in memory could easily result in heavy swapping, followed by the OOM killer kicking in. We store it on disk even if it's configured to have a finite size.
The file is unlinked immediately after creating it, so it's freed up in the unlikely case that gnome-terminal (or another VTE-based terminal) crashes. During normal operation, the file handle is released (and thus the file contents get freed up) and encryption key is explicitly zeroed out in the memory when the terminal tab/window is closed.
We used to store it in unencrypted file, but people were worried about privacy / data leakage issues e.g. if the computer gets stolen, see e.g. a generic security report and the tracking bug. (A nice extension would be if we could detect if the file system is encrypted and then skip this step.)
Konsole, on the other hand, warns you in its config dialog that "When using [unlimited scrollback], the scrollback data will be written unencrypted to temporary files." If fixed scrollback is configured, Konsole stores it in memory (and again, informs the user about it and the corresponding risks).
As an easy first step is to allow a rightclick on the tab and have an option there for scrollback lines that would just apply to that tab. Then it's all in memory and fully up to the user to choose a reasonable amount (some in memory compression might be nice but likely optional). It gets around all the security issues, and other issues if you default it to every terminal that are left open for weeks - you could auto-change the tab color so they're aware that it's different than normal.
This is a massive change.
The TextBuffer is limited to COORD coordinates which they are of short type (32767 max). Only to enable "virtual" coordinates of the TextBuffer means to change more than half of the code.
So I guess the first approach would be to change COORD to til::point, SIZE to til::size, and RECT & SMALL_RECT to... well good question. SMALL_RECT is inclusive and that could be translated to a til::rectangle. But RECT? Then you would have to call Viewport::ToInclusive() ?
It's easier to first support TextBuffer >= 32768 lines and then implement the infinite scrollback.
Buy from my perspective I think everything has to be changed, GDI renderers, UI Automation types, everything.
I don't think it should be necessary to replace the short ints everywhere (you're not going to be able to change the parameters of the legacy console APIs anyway). I reckon you could leave the in-memory buffer with a short limit, and just persist the oldest lines to disk whenever it is about to cycle. Then the only major complication is getting the scrollback viewer to page that data back into memory when the user scrolls past the top of the buffer. That's got to be easier than trying to rewrite everything with new types.
I reckon you could leave the in-memory buffer with a short limit, and just persist the oldest lines to disk whenever it is about to cycle.
You still need the virtual position to be at least a LONG (but a til::point will be better). The Viewport "moves" around this TextBuffer using these coordinates. Even if the buffer is not complete in memory and it is on disk, you need to know where in the buffer you are. The resulting screen has to be SHORT since like you're saying it is calling an API that only accepts that.
That still means changing everything except probably GdiEngine.
You still need the virtual position to be at least a LONG
But that's the only thing that needs to be LONG. The in-memory buffer really just needs to be the size of the visible viewport, with the relevant section of the on-disk buffer loaded into it.
Anyway, this was just a suggestion. Whoever is actually implementing this feature will no doubt have their own ideas on how to approach the problem. I just wouldn't recommend trying rewrite the entire conhost framework unless it was absolutely necessary, and I don't think it is.
Infinity is a really big number - personally I would love to see a 10x or 100x increase, from the current 32767 max! And/or a reference to 32767 in the https://docs.microsoft.com/en-us/windows/terminal/customize-settings/profile-advanced#history-size documentation. And/or an max value validation in the historySize field under Advanced Settings.
yea, the way putty does it works very well, its default of 2000 lines and then if you need more you can set it higher I think I've put 2M in there before I know with certainty 200,000 works fine which is plenty. Regarding security of disk writes, I think if the windows not open the file not existing would be fine but I'm not a security expert, if it's an issue having it be encrypted shouldn't be much of an additional step as only that window would need to understand how to unencrypt.
I agree that this is an extremely useful feature that every proper terminal should have. Leaving a long verbose process running just to come back and see that most of the output was lost is very frustrating.
for the love of god please fix this. You've provided an amazing terminal (finally comparable to linux apps) and then make it next to useless with no external logging or ability to have unlimited history (yes surely 'unlimited' is not that difficult to do in 2022). Back to Linux i suppose.....
Hi, this issue still exists.
Tagging @DHowett-MSFT @carlos-zamora @zadjii-msft
Yep. We'll make sure to update this thread when there is an update to share. In the meantime, might I recommend the Subscribe button?
That way you'll be notified of any updates to this thread, without needlessly pinging everyone on this thread ☺️
Ah, I see the latest comments by me, @CCCAUCHY and @zadjii-msft have been marked as "spam" and "off-topic". Very nice.
Yep. There are plenty of people following the thread for updates. Just commenting stuff like "are there any updates", "this still hasn't been added", etc, sends everyone who's following the thread a notification, but doesn't really contribute to the discussion of how to actually implement this. I'm pretty liberal with collapsing comments to keep discussions on-topic.
Yep. There are plenty of people following the thread for updates. Just commenting stuff like "are there any updates", "this still hasn't been added", etc, sends everyone who's following the thread a notification, but doesn't really contribute to the discussion of how to actually implement this. I'm pretty liberal with collapsing comments to keep discussions on-topic.
Ah, I didn't realize it was you who collapsed even your own comments. That is fine.
If there's a way to ping only MS maintainers without affecting all other readers of this, please let me know. Otherwise, I'm afraid that it will keep happening as people lose more time hitting this issue over and over.
Hi @zadjii-msft, are there any updated for this feature? Do we have an ETA? Thanks.
There's no ETA and no updates. If you subscribe to this issue, you'll be notified once that changes. Work towards this feature is currently ongoing, but it will take a very long time to finally get there.
There's no ETA and no updates. If you subscribe to this issue, you'll be notified once that changes. Work towards this feature is currently ongoing, but it will take a very long time to finally get there.
Thanks for the response. I'm subscribed to the issue, but thought I'd ping it anyway to see if there's any activity.
Work towards this feature is currently ongoing, but it will take a very long time to finally get there.
Thanks for the response. Is this work being done on a public branch of this repository? It would be nice to be able to keep track of it, since it's such an essential feature.
As a workaround, you can try using vtm to instantly get this feature (an arbitrary large ring-buffer in memory with incremental growth to a fixed maximum) inside Windows Terminal.
Run pwsh with a scrollback buffer ready to hold, for example, up to 1M lines (this will take up some RAM):
vtm.exe --tui --config "<config/term/scrollback size=1000000 growstep=100000/>" --run pwsh
https://github.com/user-attachments/assets/e3791a86-8c06-4232-9108-a5d353f30edc
You can run it in a separate window using -g/--gui cli option to see the scrollback buffer state in real time if you need to.
vtm.exe --gui --config "<config/term/scrollback size=1000000 growstep=100000/>" --run pwsh
As a workaround, you can try using vtm to instantly get this feature (an arbitrary large ring-buffer in memory with incremental growth to a fixed maximum) inside Windows Terminal.
Run
pwshwith a scrollback buffer ready to hold, for example, up to 1M lines (this will take up some RAM):vtm.exe -c "<config><term><scrollback size=1000000 growstep=100000/></term></config>" -r vtty pwshPowerShell.2024-07-20.21-34-38.mp4
You can run it in a separate window using
-g/--guicli option to see the scrollback buffer state in real time if you need to.vtm.exe -g -c "<config><term><scrollback size=1000000 growstep=100000/></term></config>" -r vtty pwsh
Thanks for the info. I haven't tried this yet, but hopefully it can serve as a workaround until Microsoft fixes Windows Terminal.
Hi, just wanted to check whether there's been any progress on this issue.
If there's a way to ping only MS maintainers without affecting all other readers of this, please let me know. Otherwise, I'm afraid that it will keep happening as people lose more time hitting this issue over and over.
You do realise you're the only person affecting this effect?
You do realise you're the only person affecting this effect?
I don't understand what you mean by this. I just thought I'd ping this to see if there's been any progress, as it's been over 5 years since it was reported and people are still tripping over it.
Is it possible for an issue as old as this one to be considered a priority and be put out of the backlog, or will it be one of those issues that will remain in backlog forever independently of how many people come here and put a thumbs up or comment on it? Is there anything we, common users can do to give it some kindo of priority? To me it affects me every day since I have to do compilations that easily pass the current scrollback limit and obligates me to use an alternative terminal, since if an error occurs it will possibly be lost due to the limited scrollback in windows terminal.
To add to that, we'll take interim workarounds as well. If the full proper fix is too involved, then at least bolt something on in the meanwhile? Cheers :p
Hi, I just got hit by this issue once again. I saw that @DHowett has set a milestone for this, does that mean there's work being done here?