egui icon indicating copy to clipboard operation
egui copied to clipboard
verified publisher icon emilk

[WIP] Render text with Parley

Open valadaptive opened this issue 9 months ago • 14 comments

  • Resolves https://github.com/emilk/egui/issues/3378
  • Resolves https://github.com/emilk/egui/issues/5233
  • Closes https://github.com/emilk/egui/pull/1687 (superseded)
  • Resolves https://github.com/emilk/egui/issues/2551
  • Resolves https://github.com/emilk/egui/issues/4364
  • Closes https://github.com/emilk/egui/issues/4311 (superseded)
  • Resolves https://github.com/emilk/egui/issues/3218
  • Resolves https://github.com/emilk/egui/issues/2517
  • Closes https://github.com/emilk/egui/pull/4644 (superseded)
  • Closes https://github.com/emilk/egui/pull/2356 (superseded)
  • Closes https://github.com/emilk/egui/pull/2164 (superseded)
  • Closes https://github.com/emilk/egui/pull/2490 (superseded)
  • [ ] I have followed the instructions in the PR template

This is very WIP and currently a proof-of-concept. I'm sharing it here so I can ask questions about the API and have something to point to if I need to merge any supporting changes.

Right now, all I have done is basic text rendering. Pretty much everything else still needs to be implemented.

You can see in the commit history that my first pass at this did use Cosmic Text. However, Parley's API surface seems to mesh much better with how egui wants to do things. I ran into a lot of limitations with Cosmic Text that will have to be solved before it's ready for this use case:

Cosmic Text issues
  • Lack of documentation for many items.
  • Lack of support for inline boxes, needed to implement things like leading_space in LayoutSection.
  • Blank lines always use the line height from the metrics on the Buffer, not the ones in any particular text span. This means that in e.g. the EasyMark Editor example, where there's mixed-size text, you need to pass the "default" font all the way down, and even then, you might still want different blank lines to have different heights (https://github.com/pop-os/cosmic-text/issues/364).
  • Setting properties on Buffer triggers one relayout per property being set (https://github.com/pop-os/cosmic-text/issues/280).
  • I never really got the hang of the "scroll" abstraction on Buffer.
  • Font fallback seems to be technically implemented but untested. For instance, it prioritizes fonts with a similar weight over fonts from the same family. It took me a while to realize why everything was being rendered in monospace: cosmic-text was choosing Hack Regular over Ubuntu Light.
  • The library seems overall quite allocation-heavy (https://github.com/pop-os/cosmic-text/issues/308).
  • No built-in AccessKit support. (I assume it's possible to implement this on top of Cosmic Text since iced does so, but Parley has this built-in for the text editing APIs)

There are some drawbacks to Parley as well:

  • They have a very aggressive MSRV policy--they can bump it at any time, including in patch releases. This means we need to pin the version of Parley we depend on. They also recently converted the codebase to Rust 2024, meaning it requires Rust 1.85 or newer.
  • Parley incorporates large amounts of Google Fonts' text stack. Large tech companies, and Google especially, tend to prioritize their own use cases above others'. They have the right to do so, of course, but it could still mean a lot more work for us.
    • Although Swash now incorporates parts of the Google Fonts stack (skrifa) as well, so Cosmic Text would also leave us dependent on Google.
  • Shaping is not completely implemented yet (it uses Swash's implementation).
  • It also lacks some features that will need to be contributed (mentioned in parley-todo.md).

There is no guarantee that I'll finish this PR; if you ping me and I respond with something like "sorry, I'll get around to it next week" a few times in a row, feel free to take over it yourself.

I'm tracking progress in more detail via parley-todo.md in the root folder, but as an overview, this work can be split into three main parts:

  • [ ] Rendering
    • [x] Implement very basic rendering
    • [ ] Implement all rendering features (e.g. ellipsis truncation, max_rows)
    • [ ] Rework the glyph texture atlas API
  • [ ] Editing
    • [x] Retool the existing text editing API to look more like Parley's (e.g. we probably don't need three cursor types)
    • [x] Abstract away parts of the text editing API that reach into the internals
    • [x] Swap it all out for Parley's API
    • [ ] Ensure that AccessKit integration works
  • [ ] Styling
    • [x] Reimplement FontDefinitions on top of Parley
    • [ ] Rework the font API since Parley supports a pretty broad range of features

valadaptive avatar Mar 13 '25 00:03 valadaptive

Regarding the risk of diverging use cases for the upstream google-funded components (fontations in particular): the use case includes basically everything people want to do with fonts, because these libraries are intended to serve the needs of web browsers (Chromium in particular, which exposes more font/shaping functionality than basically any other application) in addition to several less rich use cases.

Swash itself is likely, in time, to be replaced in Parley with RustyBuzz/Harfruzz (when the port to fontations from ttf-parser is done), and HarfBuzz (along with its ports) is again intended to cover the overwhelming majority of shaping use cases, though conceivably could fall short on very exotic shaping requirements (e.g. shaping byzantine music notation or con-scripts).

Thank you for taking on this task. It is exciting to see interest in adopting Parley, and your work has already shaken a lot of dust out. :+ )

xorgy avatar Mar 17 '25 22:03 xorgy

This is very exciting - thanks for working on this! Let me know if I can help somehow, e.g. if you want to talk things through.

emilk avatar Mar 18 '25 14:03 emilk

They have a very aggressive MSRV policy--they can bump it at any time, including in patch releases. This means we need to pin the version of Parley we depend on. They also recently converted the codebase to Rust 2024, meaning it requires Rust 1.85 or newer.

Thanks for the feedback!

I've just landed a revert of that in https://github.com/linebender/parley/pull/307 so now Parley's next release will still have an MSRV of 1.82 as the current release does.

waywardmonkeys avatar Mar 20 '25 01:03 waywardmonkeys

I'm now mainly working on the text styling API. It's a bit hard to untangle everything, since there are several different places where text style properties are stashed, each exposing a different API surface:

  • FontId is what lets you select a specific font and size.
  • TextFormat contains a FontId as well as some formatting-specific things like color, background, italics, and decorations.
  • TextStyle associates names to FontIds (either predetermined names like TextStyle::Small or user-defined ones via TextStyle::Name(_)).
  • FontSelection is either a FontId or a TextStyle.
  • RichText is kinda like TextFormat except with an override/fallback mechanism? When converting one into a TextFormat, you pass in a "fallback font" (a FontSelection), but if either the RichText's text_style field or the containing UI's Style::override_text_style are set, they will be used to select the font instead.

Parley unlocks some new options for font styling, and also changes the semantics of font styling a bit:

  • Instead of a single "font" like in FontId, you specify a font family. Depending on style properties like weight, italics, etc. this may be resolved to one or many actual font files.
  • There are more styling options--"real" italics if supported by the font family, font weight, font width (if supported), and OpenType font variation settings.
  • Instead of specifying just one font family, Parley's APIs actually take a FontStack, which allows for font fallback.

I ran into some issues when trying to map the existing text style API onto Parley. First up, which new font properties should be part of FontId? My initial thought was anything that affects layout, so that would be font family/stack, size, slant, weight, width, and OpenType variation and feature settings.

However, italics are currently in TextFormat and make sense there. Does that mean that weight should also be in TextFormat? But the default UI font is Ubuntu Light. Maybe specify a default font weight as part of FontId and allow TextFormat to "override" it?

If we take that to its conclusion, why not just flatten all the font settings then? Have one big "all of the font style settings" type, and make it overridable CSS-style. I'm not sure yet what this API would look like exactly.

I'm also not sure which parts of this new API I can punt on right now, and which ones must be implemented.

valadaptive avatar Mar 23 '25 16:03 valadaptive

My inclination is to punt on as many things as you can, and save it for a future PR. Better to break things into small pieces.

The existing API is indeed all over the place, and partially a result of legacy, and partially a result of technical limitation (e.g. the fake italics). Worth mentioning is that font fallbacks is currently implemented in egui here.

Moving italics into FontId makes sense to me, as it is about selecting a specific font. As does adding weight to it at some point.

The rest of the attributes in TextFormat should probably stay there, as they are about how to apply that font.

emilk avatar Mar 25 '25 08:03 emilk

Things are progressing well; just need to land some Parley changes and clean up the code. There are some issues that might change the API more significantly:

  • Galley needs to be serializable via serde, but Parley's types currently are not. I theoretically could implement serde support for things like parley::Layout, but I'm wondering if it even makes sense--there are a lot of things like font family IDs that are based on auto-incrementing counters and would thus not be meaningful through a serialization round-trip. This is sorta already an issue with the current code--AFAIK, the text meshes themselves reference UV positions in the font atlas that are also not guaranteed to be stable for any length of time, let alone a serialization round-trip. Is there a serialization behavior that makes sense here?

  • LayoutJob::break_on_newline is probably not implementable without some changes to Parley. Is there an alternative behavior that would work for singleline TextEdit?

  • FontTweak::scale says it does not affect the layout, but it does affect the advance width of the characters. Parley lets you change the font size for a given bit of text, but because it lets you select multiple font families, there's no way to know what font a given character will actually fall back to, and hence no way to adjust the font size ahead-of-time to match. Is it OK for it to not affect the characters' advance width?

Also, a question: is there any actual difference between FontTweak::y_offset_factor and FontTweak::baseline_offset_factor? Neither appears to actually affect the layout, and from playing with the "Font Tweak" UI in the settings, they both appear to do exactly the same thing. FontTweak::baseline_offset_factor was added in https://github.com/emilk/egui/pull/2724; maybe the behavior has changed since then?

valadaptive avatar Mar 30 '25 01:03 valadaptive

  • Galley needs to be serializable via serde

I agree with you here - let's drop serde-support for Galley. I was using it for eterm way back, but at the moment it makes little sense (as you point out).

  • LayoutJob::break_on_newline is probably not implementable without some changes to Parley. Is there an alternative behavior that would work for singleline TextEdit?

To explain break_on_newline, consider this code:

let mut start_text = String::new("Already \n multiline");
ui.text_edit_singleline(&mut start_text);

How should egui handle the existing \n, which clashes with the desire of the programmer to have the TextEdit be single-line? The answer right now is that egui sets break_on_newline: false which renders newlines as the replacement glyph, �.

Possible alternative solutions include: A) Replace any '\n with � before passing the string to Parely (if break_on_newline == false) B) Have singeline TextEdits replace any \n with � in the input buffer (mutating the users string) C) Ignore the problem (remove break_on_newline and say that having newlines in the text buffer of a TextEdit will just lead to weird behavior, so "don't do that"

Since this is such a niche corner case, I'm happy with any of these solutions.

  • FontTweak::scale says it does not affect the layout, but it does affect the advance width of the characters. (…) Is it OK for it to not affect the characters' advance width?

Yes. FontTweak::scale is currently used to make the sizes of the emojis in the different default fonts match up, so that when setting fotn height =12 pts (for instance), the emojis look the same (even though one of the fonts have big emohis, and the other small ones). I don't think advance width matters. Do what feels best :)

Also, a question: is there any actual difference between FontTweak::y_offset_factor and FontTweak::baseline_offset_factor?

These were added to match the baseline of normal text and emojis (iirc). As long as we can still do that reasonably well, do whatever change you want 👍

In summary: I'm fine with this breaking API, behavior, and rendering in small ways. The wins will outweigh those minor annoyances imho!

emilk avatar Mar 30 '25 11:03 emilk

@valadaptive btw, let me know if I can help by talking things through on Discord with you!

emilk avatar Apr 01 '25 14:04 emilk

Discord would be great! My username there is the same as here.

I see that #5411 just got merged--that's going to involve a major rearchitecture of the work so far, possibly to the point of creating a new branch and manually redoing however much of what I've done so far is still relevant.

It may take a while to get around to that; I'm working on some upstream Parley stuff right now.

valadaptive avatar Apr 01 '25 21:04 valadaptive

Preview available at https://egui-pr-preview.github.io/pr/5784-parley-2 Note that it might take a couple seconds for the update to show up after the preview_build workflow has completed.

github-actions[bot] avatar Jun 12 '25 11:06 github-actions[bot]

Was excited to learn about this PR today! (bold! better italics! Color emoji! variable fonts! ❤️❤️)

I briefly tried out the preview here and saw a minor issue w/ rendering source code links: https://egui-pr-preview.github.io/pr/5784-parley-2

Screenshot 2025-08-11 at 9 42 49 AM

If there are any pieces you can think would be easy to independently tackle to unblock this, I'd be happy to help. I have the same username in the Discord if you'd like to ping me there.

NfNitLoop avatar Aug 11 '25 16:08 NfNitLoop

https://github.com/linebender/parley/releases/tag/v0.6.0 :eyes:

Parley now uses HarfRust rather than Swash. This means that Parley now has production-quality shaping for all scripts and can be recommended for general usage.

torokati44 avatar Oct 06 '25 13:10 torokati44

An update on this: I was supposed to receive funding to work on a rework of Parley's API to make it more suitable for this project and others like it, but that never materialized. At this point, I'm more likely to just go directly to the underlying APIs (skrifa, HarfRust, and whatever rasterizer I end up going with) and wire those into egui.

To an outside observer, it probably looks like this PR is 90% of the way there and just needs that last 10%, but unfortunately that last 10% is basically impossible considering how Parley's API is structured. It really wants to be given a full rectangle that it can arrange text inside, however it wants, and egui really doesn't work with that model. This PR uses a lot of workarounds to try to hide that, but they aren't reliable. There have been a few Parley proposals to make its API even more convoluted in order to support some of this stuff, but I have doubts about those working for egui.

valadaptive avatar Nov 10 '25 03:11 valadaptive

@valadaptive

Thank you for working on this.

Mentioning/Pinging this PR when/if your new approach starts materializing would be appreciated, so those of us watching from here can follow along.

I'm holding off trying GUI in Rust, despite using it for almost everything else, because I want to experiment with all the big-name crates first, and fancy text shaping is a requirement for me.

BergmannAtmet avatar Nov 13 '25 15:11 BergmannAtmet