Subpixel font rendering

[20k]

Custom fork - https://github.com/20k/imgui-fork https://github.com/20k/imgui-fork/commit/3f01a60df266c551f36d41dec3c7c141ae50b25a specifically (ignore the commit before)

SFML + opengl + windows 10 + msys2

ImGui-SFML - custom fork https://github.com/20k/imgui-sfml-fork

Hello there! I've been using ImGui for a long time now. One of the main problems with it is the lack of subpixel antialiased font rendering. This was frustrating enough that I wrote my own implementation, so here it is. I would very much like to get this integrated into ImGui proper, however there are some caveats and tradeoffs that need to be detailed here

1. This turns imgui's colouring linear. I assume that simply running a conversion on all imgui's colour styles and linearising them is fine, with no further changes. On the plus side this seems to work pretty much fine, although it does change window dropdown lists to be a bit more transparent. I also make no effort to store the original colours, so there is a very small precision loss if you keep swapping from linear to srgb colour spaces. It may not be appropriate to apply this transform to the alpha component, so I'll have to investigate

2. You need an srgb framebuffer for this to render optimally, although its not necessary. On linux under wine I have noticed problems with an srgb framebuffer, resulting in very poor colouring

3. To properly blend subpixel fonts you need to modify the rendering backend, in particular you need to use the extension https://www.khronos.org/registry/OpenGL/extensions/ARB/ARB_blend_func_extended.txt here. I have no idea whatsoever what hardware or backend support for this is like

4. To get the most accurate subpixel font rendering when rendering coloured text, you need to use a shader. This only provides a very minor increase in rendering quality for quite a bit of performance. You can avoid this with no real penalty but colour pedants might mind

5. Even in the worst case of non linear colour on a non srgb background this still significantly increases clarity of text

6. I use SFML internally within the implementation to store the font atlas. Obviously this ties it to OpenGL - I am not familiar enough with ImGui to know quite where to go with this

7. I have copypasted some code (mouse cursor definition, parts of font atlas building) into freetype. I'm open on how to fix this

8. I believe freetype needs updating as well

I have mostly cleaned up the code under imgui-fork, and I have not cleaned up the code under imgui-sfml-fork as that's a task for another day. Imgui-fork is based off a relatively old version of imgui now

If you're keen on this i'll do some work on this to fix everything up etc

Positives:

1. Massively better font rendering

2. Supports all the different filtering modes

3. Not much performance overhead that I can see

No subpixel AA (bitstream vera sans mono):

Legacy filtering subpixel AA (seems to be best but its configurable) no options set

Legacy filtering subpixel AA with forced auto hint

Coloured text rendering no subpixel AA:

Coloured text rendering with subpixel AA and forced autohint

These previews kind of butcher it so I'd recommend opening them in paint or something

No spaa: forced autohint

spaa: forced autohint

[20k]

New version rebased on master https://github.com/20k/imgui-fork/commit/c115d9ea9e6738a27eb91d008d655322db6a0b35

This fixes the dependency on SFML - the freetype class takes a pointer to a function pointer to write the data to the GPU. This is basically necessary because imgui internally stores fonts as 8 bit grays, whereas this needs at least 24bit full rgb, so writing the data early sidesteps this

It doesn't support the multiplication table yet but it should be straightforward once I figure out what the multiplication table is

This version also fixes some of the copypaste job as now it exports the mouse cursor/ascii art stuff. However it still reimplements part of ImGUI's logic for finishing a font build, because that only supports 8bit

[ocornut]

Hello James, Sorry for not answering much to this. I may not have time to dig in details but this is really useful.

It doesn't support the multiplication table yet but it should be straightforward once I figure out what the multiplication table is

It's basically a little helper to increase brightness. There's not really much correct science to it (we linearly multiply 8-bit values to in srgb space etc.) so it isn't advertised much, but with current rasterizers and small resolution fonts a boost of a few % tends to helps readability by making glyphs appears sharper. I added it because Michael Sartain used it in gpuvis (along with FreeType). From the user point of view it is a solely a float multiplitier, internally turning it into 8-bit lookup table is probably overkill there, but may come with the added benefit that we could handle correcter linear vs srgb multiply with more ease by generating different tables.

This is basically necessary because imgui internally stores fonts as 8 bit grays, whereas this needs at least 24bit full rgb, so writing the data early sidesteps this However it still reimplements part of ImGUI's logic for finishing a font build, because that only supports 8bit

I'd be open to reworking some of those internal API, if 32-bit RGBA is desirable earlier in the pipe. In fact, a PR solely dedicated to reworking some of the internals with the purpose of facilitating mods such as this one are welcome. It would be faster and easier to merge such PR with limited purpose/scope first, than the larger feature which may require more work and iteration.

Note that what you implemented would also work in stb_truetype land just as well (using X3 horizontal sample as a starting data for the data). I understand that if the user is aiming for that higher quality provided by subpixel they'd probably want to use FreeType in the first place. But it might be sane to exercise the design and internal api (the changes mentioned in paragraph above) by having this also work with stb_truetype.

[20k]

I'm open to implementing a rework of the internal API's, TexPixelsAlpha8 does not return too many results and generally seems to be used internally, and I've already got some idea of how it's used

Simply swapping out everything that uses TexPixelsAlpha8 with a 32bit version should be sufficient - however to be useful for subpixel font rendering, additional changes must be made to the way that imgui issues draw calls so that text can be isolated out and rendered correctly, eg https://github.com/20k/imgui-fork/commit/c115d9ea9e6738a27eb91d008d655322db6a0b35#diff-ce7fa9ef5b9e7238e581282e6948c9bdR2976

I can either submit these as two requests (rgba8 -> rgba32) + (draw call/submission rework) or one together (rgba8 -> rgba32 + draw call/submission rework) once I've completed the former

It would be easy to get compatibility with stb_truetype initially by copying the rgba-8 data to the new rgba-32 pointer (with appropriate conversions depending if subpixel rendering is enabled or not) which I'm happy to do. However, it is not enough to render triple width to get alright subpixel font rendering (as filtering and weighting during rendering needs to happen) which would realistically mean digging into stb_truetype as it has specific subpixel rendering modes. It would likely be better quality to convert 8 bit grays into 32bit coverage (so in practice the same font rendering as before) in the short term as colour fringing is a bit painful to look at after a while, until someone wants to implement full subpixel rendering through stb's apis

Just for reference: No filter (freetype because I don't have an stb pipeline on hand to give a more precise example):

No subpixel AA (freetype):

[tomasiser]

Hi @20k,

I am really excited about your subpixel font rendering!

I am no expert in this field, but I know that subpixel rendering is especially problematic with various backgrounds and transparency, and afaik certain subpixel implementations including ClearType are disabled on transparent buffers by default [1] [2] [3] [4]. Similar problems happen with high-DPI scaling where subpixel rendering does not work properly when scaled.

You seem to understand subpixel rendering quite a lot, so I wanted to ask if it will be required to build a new font atlas for every background color or if your implementation is independent on background and it works correctly with semi-transparent windows (such as pop-ups).

Good job on finally bringing subpixel rendering to ImGui!

[20k]

Rendering subpixel fonts is outside the purview of the pull request that I'll submit to imgui, as imgui delegates actual rendering out. That said, in my implementation of the rendering, there is no reason why it won't work on coloured backgrounds or with transparency (or with the text being coloured itself, though there are caveats there), as shown in the picture below. I don't know anything specific about the challenges of high dpi, all I'm doing is taking a set of rasterised pixels and blitting to the screen, taking care to make sure that blending is done linearly

Rendering it in this fashion involves a dual blending source shader or a regular shader + glBlendColor or similar. There's an issue in that avoiding dual source shaders means more draw calls, but imgui has to know whether or not you want glblendcolor compatible behaviour or shader behaviour (purely for performance reasons)

There's a tradeoff with rendering coloured text to do with brightness accuracy vs shader complexity, eg a pixel coloured red that only has the left 1/3rd covered (ie the red channel) needs to be decreased in brightness appropriately, which is outside the normal expectation of vert_col * tex_col

So overall tl;dr no font atlas per background, works fine with popups, then i need to merge some draw calls to alleviate performance issues, and then do a lot of checking and tests to ensure that it all works correctly. The quality of the rendered text will depend on the provider of your window and rendering backend getting everything correct, which is not a given as its hard to do correctly

[tomasiser]

The screenshot looks nice. As I said, I do not have enough knowledge to understand the issues with subpixel rendering in details, but on your screenshot I cannot spot any artifacts and the font looks definitely much clearer than what we usually get without subpixel rendering.

What I was refering to previously is the fact that some implementations / browsers have troubles with subpixel rendering in some cases and I do not yet understand why and if it has anything to do with possible artifacts in some special cases. For example in Spotify, some parts of the GUI are rendered with subpixels, some are not (zoom in the following screenshot):

[20k]

I don't have enough knowledge to give you a complete answer there. Some of it is probably implementation difficulties (its hard to do correctly), some of it is performance concerns (requires shaders or opengl extras to blend correctly, extra draw calls), some of it is that it may simply not be worth it in the high dpi case (resolution is already good enough), there are concerns over patents in the field, concerns over how effective subpixel font rendering really is due to varying human colour perception, and additionally subpixel font rendering as a technology is very old, and implementations are saddled with that as well (non fixed colour backgrounds are hard to do on old hardware). I don't know specifically why spotify works like that

None of this should be a dealbreaker in this case however - implementation complexity is less of a concern here because its optional and a lot of the complexity (linear colour pipeline) will be part of this patch - the main problem currently is how to decide on which model to force on the rendering backend (fewer draw calls with complex shaders, more draw calls with simpler shaders)

[ocornut]

Just a quick rection:

Rendering subpixel fonts is outside the purview of the pull request that I'll submit to imgui, as imgui delegates actual rendering out.

I think we ought to have support in at least the common renderers (e.g. GL, DX11) so people (me included) can test and evaluate and work on the feature eventually. However we design the change is an open question, but don't hesitate to share some of the implementation code as well.

[20k]

I think we ought to have support in at least the common renderers (e.g. GL, DX11) so people (me included) can test and evaluate and work on the feature eventually. However we design the change is an open question, but don't hesitate to share some of the implementation code as well.

I'm happy to implement this for OpenGL as I've already done much of something similar over at the imgui-sfml fork (based on an old version of imgui-sfml), but my knowledge of DX is very limited. I do have a background in 3d rendering but its largely gpgpu/opencl, so I'm not convinced that my opengl (or potential directx) implementation will be optimal at all - so if there's someone about who knows about this it would definitely be worth a look at the final code to see if there's any improvements that can be made. Particularly I don't know what the state of hardware compatibility is like for anything

Just to check the specifics, are you talking about updating an example, like imgui_impl_glfw.cpp and example_glfw_opengl3?

For testing purposes I'm planning to have an easy project https://github.com/20k/imgui_font_tester to put everything together, and test different fonts, and also to demonstrate stb vs freetype vs subpixel AA freetype on differently coloured backgrounds etc, although everything is still WIP

[Nielsbishere]

The fonts still look blurry to me. When sampling the texture atlas (R8), you should gamma correct it like mentioned here http://www.puredevsoftware.com/blog/2019/01/22/sub-pixel-gamma-correct-font-rendering/. That will make the fonts look sharper.

[20k]

This fork (optionally) gamma corrects using sRGB framebuffers https://github.com/20k/imgui-fork/commit/5a336d30c26d5645fea2bca56b0fac65f3af799c instead of performing the conversions manually in shaders like in the article

No gamma correction:

Gamma correction using an sRGB framebuffer:

Its worth also noting that the gamma correction that that article performs, aka pow(col, gamma) and pow(col, 1/gamma) is incorrect, though widely used. See here for more details

There probably are subtle issues with the placement of glyphs (ie subpixel positioning), and blending (white on black doesn't seem to come out too great with gamma correction, which is on my todo list to investigate)

It does look much better than baseline freetype with no subpixel AA to my eyes without much blurring, but given that this has all been tested on my own monitor and largely white on black, I may have missed something! Is there a specific example you think looks blurry out of interest?

[Nielsbishere]

I was on mobile, it looks better on pc. However, in the article he does explain that gamma 2.2 (which is what sRGB uses IIRC) is too much correction and means your outlines will become a lot less thick (so less benefit to subpixel rendering). In the case of fonts, the gamma correction isn't exact, so the pow(gamma, 2.2) wouldn't be that, because they made fonts thicker for people who don't gamma correct. However it is interesting for other cases like colors. The gamma corrected one does look clearer tho :+1: It might be handy to test if it looks correctly if you rotate your screen (since you have to use vertical subpixel rendering instead of horizontal) or try it on a different screen, since subpixel rendering varies heavily on the screen you're on. There's a lot of layouts it doesn't work with (RGBW, RGBY, RGBYC, RGBG, some mobile layouts like the iPhone X (PenTile)) and other layouts such as BGR screens need modification as well. EDIT: Mmm, I added it to my own application and actually it doesn't seem like it matters much (RGB/BGR and rotation)

[Nielsbishere]

This is with gamma 1.43:

This is with gamma 2.2:

1.43 looks fuller and less transparent, more readable imo.

[ocornut]

Also see Freetype Subpixel Rendering #2986 PR by @loicmolinari

[20k]

So, this is the freetype backend which only supports two hardcoded subpixel modes: LCD, and LCD_V basically, which are both exposed in the API for this fork https://github.com/20k/imgui-fork/blob/master/misc/freetype/imgui_freetype.h#L28. Rotation won't work automatically

Freetype can be made to support Pentile and BGR by using custom subpixel layouts with some effort, but none of this is part of this fork, as it just exposes the hardcoded freetype rendering modes. I don't have any devices that I could test subpixel layouts on with ease - but if someone forks it in or gives me a patch I'll happily merge it into this fork, although I have plans to replace this fork with one which is more merge friendly for imgui master

That article is somewhat misleading as there's a crucial component missing to the 1.43 vs 2.2 discussion, which is what LCD filter is used. With the old freetype filters (FT_LCD_FILTER_LEGACY) - you are right, they were designed for applications which do not correctly handle gamma and is a compromise filter, but the modern filter (FT_LCD_FILTER_DEFAULT) specifically is designed to work with gamma aware rendering. Even if it were correct, the equation used is still wrong, as pow(val, 1.43) does not do what you think it does, and you need to use the real sRGB equation (or a closer approximation)

Hinting is also super important, the following is illustrative to show that you can't really just say that 1.43 vs 2.2 is a key differentiation without more information

Force auto hint, modern filter, gamma correction:

No auto hint, default filter, gamma correction: (wildly different quality)

Force auto hint, legacy filter, gamma correction: (notice the colour fringing)

Force auto hint, default filter, no gamma correction: (font is too dark, colour fringing)

Force auto hint, legacy filter, no gamma correction: (colour fringing)

So more information required essentially

There are a few additional problems here:

1. sRGB framebuffers (big performance win) are only sRGB (aka 2.2 gamma), which means that it would be difficult to use a 1.43 colour space even if it were a net positive (otherwise I'd post results!). The alternative is to perform gamma correction in a shader, which is significantly slower because this fork relies on dual source blending to be done in the correct colour space

2. Using different gamma for fonts and regular drawing is confusing. Font rendering is a series of compromises at the best of times, but rendering red font on a red background should produce precisely the same reds

[20k]

I'll have a look at that PR though!

[Nielsbishere]

Good points, makes sense. So these 3 values can also be modified to support vertical bgr, rgb and pentile. I've always wondered how this would be detected though; what screen type you have. Windows has a way to query the dmDisplayOrientation of the current monitor (or monitors if you really want to do it 100% correctly) but they don't have support for all native formats. And I know with vulkan you can query the real (BGR or RGB) format of the backbuffer, not sure about opengl, still looking for it. However, that's probably out of scope Edit: it seems like you can also query if the screen is BGR or RGB through DWrite (windows specific)