3D Fluent Emojis

[tusharsnx]

Description of the new feature

Windows added 3D Fluent emoji support in 23H2. Supporting 3D Fluent emoji in Windows Terminal would be a nice addition.

Proposed technical implementation details

3D Fluent emojis are implemented in COLRv1 format. We'd have to support drawing DWRITE_GLYPH_IMAGE_FORMATS_COLR here:

https://github.com/microsoft/terminal/blob/bd7e3179ff1c41e9147d99253a9ab150cf9e942a/src/renderer/atlas/Backend.cpp#L73-L105

Resources:

https://learn.microsoft.com/en-us/windows/win32/directwrite/color-fonts-preview#layers-of-solid-color-glyphs https://learn.microsoft.com/en-us/typography/opentype/spec/colr#colr-version-1-rendering-algorithm https://blogs.windows.com/windows-insider/2023/07/12/announcing-windows-11-insider-preview-build-25905 https://learn.microsoft.com/en-us/windows/win32/api/dcommon/ne-dcommon-dwrite_glyph_image_formats

[JohnnyWestlake]

If it does get added, make it optional please. They're extremely expensive to render compared to the COLRv0 variants.

[tusharsnx]

Atlas engine caches all the glyphs it renders, so you are only really paying the rendering cost on first use of any glyph. Later rendering just copies the cached glyph, which is extremely fast. I do have to confirm whether we cache emojis output or not. @lhecker

[lhecker]

I don't want to add support for COLRv1 emojis until their performance is fixed. Notably, they're not actually expensive to render at all! You can easily render all of them at 1000s of FPS. So, this is purely due to the current implementation in Direct2D. We've been waiting for this to be fixed.

Aside from a subjective refusal there's also a real, objective problem: The only API that Direct2D offers to get the pixel size of a glyph is ID2D1DeviceContext::GetGlyphRunWorldBounds. We need that API to allocate sufficient space in the glyph atlas. MSDN fails to mention that this API was written for Windows 7 and was never updated to properly support Emojis since then. If you use an SVG emoji font with Windows Terminal, it'll fail in funny ways because of this. The only reason it appears to work is because COLRv0 emojis are basically just colored layers of regular TTF glyphs which are supported by that API. COLRv1 emojis thus cannot be supported either, because just like SVG emojis, their rasterized size information is bogus. We're similarly waiting for a fix on that.

[lhecker]

Later rendering just copies the cached glyph, which is extremely fast. I do have to confirm whether we cache emojis output or not.

Oh, to clarify: Yes, we cache these glyphs, but only for the D3D renderer. But that doesn't fix the severe hitches you see when the COLRv1 emojis first appear on the screen. We also still have an important D2D renderer which is used when accessing remote servers without discrete GPU.

[DHowett]

Considering the discussion here, I think we'll probably be closing it Won't Fix + Tracking-Internal... for now.

[tusharsnx]

@lhecker, what you said makes sense; we do need glyph bounds in advance but GetGlyphRunWorldBounds doesn't work well with SVG/COLRv1. I actually saw MSEdge and Chrome supporting the 3D emojis, which made me believe it's stable enough to be implemented in other places. It seems I was wrong 😅. I also checked how they determine the bounds for the glyphs, and it appears they're using Skia for that.

[lhecker]

I also checked how they determine the bounds for the glyphs, and it appears they're using Skia for that.

I believe they also use Skia for rendering them. To test this, you can use Unicode's emoji-test.txt. I've also created a variant with all emojis stripped for comparison: emoji-test_stripped.txt

If you open these files in Notepad and scroll through them, you'll see severe hitching. I recommend pressing and holding PageDown for instance, to make it consistent. If you try it with Chromium on the other hand it'll be roughly an order of magnitude better, even though it rasterizes on the CPU. It'll still hitch, however. I believe this is because Skia also doesn't come with specialized COLRv1 implementation but rather builds it on top of the existing gradient facilities. (This is the same for Direct2D, but it's worse, because it uses GPU rendering which adds latency on top of the issue.)

Side note: COLRv1's goal to my knowledge was to have a "mini SVG" spec for fonts. I think they definitely reached that goal and to that end it's great. What I don't understand, however, is why we then use it extensively for the one thing SVGs are really bad at: Detailed gradients. Gradient meshes would not just compress these 3D emojis better within the files but also be way easier to optimize for GPU rendering. Alas... COLRv2 has mesh gradients on the backlog but I continue to believe that the COLR specification overall is not written with the complexity of optimizing its rendering in mind.

[DHowett]

@lhecker should we close this? Or, what's the next step?

[lhecker]

The ideal solution is that we contribute improvements internally. If we want to pursue that, we could leave this open. Otherwise, I think we can close this.