Processing large data is extremely slow (in VMware)

[egmontkob]

Environment

Windows build number: Win32NT 10.0.18362.0
Windows Terminal version (if applicable): 0.5.2762.0

inside VMware Workstation Player 15.5.0 on Ubuntu 19.10

Core i5-6200U @ 2.30GHz

Steps to reproduce / Actual behavior

cat'ing my favorite test file for speed measurement takes extremely long time.

Inside VMware, in Windows Terminal it takes about 10 minutes of wall clock time.

On Linux (inside the same VMware) it takes from 2 to 22 seconds mesasured in various terminal emulators, namely: Konsole (KDE), Pterm (PuTTY), St (suckless), Terminology (Enlightenment), Urxvt, VTE, Xterm.

The test file is ~42MiB large, contains ~667k lines. It's the output of a ls -lR --color=always / on Ubuntu. (I'm not attaching it since it could leak private stuff, plus it would be a pointless waste of storage space.)

cat is executed either locally in PowerShell, or remotely over ssh to my host computer, it doesn't matter.

On the previous version of WT which I installed about a week ago, it took about 9:22 (the same time twice) to cat this file at the terminal's default size. If the terminal was iconified, or I switched to another (idle) tab, the time dropped to 6:50-ish. Interestingly, in a tiny but visible terminal (approx. 30x4) the time increased to 11 minutes. In a giant terminal (maximized with pretty small font) the time hardly increased, to 9:37.

With the current WT version 0.5.2762.0 now I'm seeing even larger numbers: 10:39 at the default size (measured only once), 7:30-ish in minimized window or when viewing another tab.

The exact times probably don't matter too much, we're talking about the magnitudes here, it's ~100x slower than VTE for example with its 5.2 seconds if it's in a good mood.

The given example is sure an extreme one (why would anyone cat such a giant file?), but smaller files, such as /etc/services already take a noticeable ~0.5 seconds, whereas on Linux terminals it's instantaneous. For verbose compilations of large projects, this could actually cause a noticeable productivity loss for developers.

I don't know whether running under VMware (e.g. no hardware graphics acceleration) is relevant, but again, the Linux numbers were also measured inside VMware, on a Fedora 30 guest.

Expected behavior

Windows Terminal should be comparably fast to most graphical terminals on *NIX.

Since I don't know the reason for the slowness (which needs to be investigated first), and I don't know whether it's specific to VMware, these are random guesses only and might completely miss the actual problem:

The terminal should read and parse as much data as possible, only stopping for updating its UI according to the monitor refresh rate, typically 60 times per second (or maybe at a hardwired 60Hz if refresh rate can't be detrmined or the concept doesn't exist – I don't know how it goes in VMware). If updating the UI takes so much time that there's hardly any time left for processing incoming data, it should start dropping frames (VTE counterpart, kind of). In iconified state or when another tab is selected, it shouldn't spend any time on drawing.

---

Note that I've checked other bugreports about slowness, e.g. #1064, but they don't seem to be about this kind of extreme slowness.

[DHowett-MSFT]

Re-marking this one for post-1.0 instead of pre-1.0. We're making improvements here, such that we can cut down on a bunch of unnecessary rendering, but we're not quite to the point where we can just shuttle all the data through the console driver at a great enough speed to make this "instantaneous". We'll keep investigating after our launch. Thanks for the robust bug reports, as always, @egmontkob.

[miniksa]

https://github.com/microsoft/terminal/commit/90a24b20b8d27edbc8451936d215cf111cfe3164 is my attempt at experimenting to see if we can make this go even faster by breaking the locking that is occurring here.

For big.txt from #1064 (which is about 6MB), I go from

real    0m3.838s
user    0m0.000s
sys     0m0.150s

to

real    0m0.124s
user    0m0.000s
sys     0m0.113s

The graphical output still takes longer than that, but it's not backing up the actual I/O. Also, it's a super dumb and rough implementation to try to prove whether this is worth pursuing. It's nowhere near ready. But I think it proves that there's a good return on investment to be had in this area by breaking up the locking.

[NeKJ]

Yes, IMO it's definitely worth it. The display/rendering should never block IO/CPU operations (as it happens now if I understood correctly).

[miniksa]

Yes, IMO it's definitely worth it. The display/rendering should never block IO/CPU operations (as it happens now if I understood correctly).

Well... I can't have it "never" block IO/CPU unless I consume an infinite amount of memory or otherwise optimize the entire pipeline to be balanced.

[Po-wei]

@miniksa Just a small finding. Using the same big.txt from #1064 with my computer. time cat big.txt took

real    11.26s
user    0.00s
sys     1.61s

But if I do a ssh localhost first and then type the same command it took

real    3.38s
user    0.00s
sys     0.17s

For the rendering, both looks identical for me. Not sure if ssh do a larger buffer in the background or something

win 10.0.19042.928 terminal 1.8.1032.0 ubuntu 20.04