WIP - Implement Universal Font Shaper

[JimBobSquarePants]

Prerequisites

• [x] I have written a descriptive pull-request title
• [x] I have verified that there are no overlapping pull-requests open
• [x] I have verified that I am following matches the existing coding patterns and practice as demonstrated in the repository. These follow strict Stylecop rules :cop:.
• [ ] I have provided test coverage for my change (where applicable)

Description

A Work-In-Progress implementation of the Universal Shaping Engine (USE). Fixes #217

There's a fair amount to do here but I have some of the groundwork laid out already. I was hoping to base the work primarily on FontKit but I have no idea how to implement the state machine.

If anyone from the community is reading this PLEASE HELP OUT. I can't possibly figure this all out on my own ❤️.

Guide. https://github.com/n8willis/opentype-shaping-documents/blob/master/opentype-shaping-use.md

HarfBuzz Implementation https://github.com/harfbuzz/harfbuzz/blob/main/src/hb-ot-shaper-use.cc

FontKit Implementation https://github.com/foliojs/fontkit/blob/master/src/opentype/shapers/UniversalShaper.js

TODO:

• [x] Refactor shapers to allow shaping stages.
• [x] Add Indic shaping data.
• [ ] Add USE specific Extended Indic shaping data.
• [ ] Implement state machine.
• [ ] Implement category lookups.
• [ ] Implement shaper.
• [ ] Add tests.

[codecov[bot]]

Codecov Report

Merging #288 (ce4f8a3) into main (7816350) will increase coverage by 1%. The diff coverage is 93%.

@@           Coverage Diff           @@
##            main    #288     +/-   ##
=======================================
+ Coverage     83%     85%     +1%     
=======================================
  Files        225     234      +9     
  Lines      12807   13922   +1115     
  Branches    1837    1917     +80     
=======================================
+ Hits       10757   11838   +1081     
- Misses      1613    1631     +18     
- Partials     437     453     +16     

Flag	Coverage Δ
unittests	85% <93%> (+1%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/SixLabors.Fonts/StreamFontMetrics.Cff.cs	90% <0%> (ø)
...e/Resources/IndicSyllabicCategoryTrie.Generated.cs	100% <ø> (ø)
...nicode/Resources/UniversalShapingTrie.Generated.cs	100% <ø> (ø)
src/SixLabors.Fonts/StreamFontMetrics.TrueType.cs	97% <50%> (ø)
...Tables/AdvancedTypographic/Shapers/ArabicShaper.cs	89% <70%> (+<1%)	:arrow_up:
...bors.Fonts/Tables/AdvancedTypographic/GPosTable.cs	86% <71%> (+1%)	:arrow_up:
...les/AdvancedTypographic/Shapers/UniversalShaper.cs	81% <81%> (ø)
src/UnicodeTrieGenerator/StateAutomation/INode.cs	86% <86%> (ø)
...Tables/AdvancedTypographic/Shapers/ShapingStage.cs	90% <90%> (ø)
src/SixLabors.Fonts/TextLayout.cs	85% <91%> (-1%)	:arrow_down:
... and 16 more

... and 2 files with indirect coverage changes

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[JimBobSquarePants]

OK.

So I have the beginnings of a plan here....

I need to port the state machine used by JavaScript from here https://github.com/foliojs/dfa

Instead of using Pegjs as the PEG (Parsing Expression Grammar) parser generator I can use Pegasus.

However... The original sample really abuses type safety so I'm having real difficulty with my early porting experiments.

[stefannikolei]

However... The original sample really abuses type safety so I'm having real difficulty with my early porting experiments.

any should go to hell xD

i was looking for other implementations besides the js and c implementation of harfbuzz.

I found a rust port: https://github.com/RazrFalcon/rustybuzz/blob/master/src/complex/universal.rs

perhaps this helps

[JimBobSquarePants]

However... The original sample really abuses type safety so I'm having real difficulty with my early porting experiments.

any should go to hell xD

i was looking for other implementations besides the js and c implementation of harfbuzz.

I found a rust port: https://github.com/RazrFalcon/rustybuzz/blob/master/src/complex/universal.rs

perhaps this helps

That rust port has an open issue because it's missing a Ragel compiled state machine, the state machine is the bit I probably need most.

I've had a look at Ragel before, but I cannot make head no tail of it. Supposedly you can build a C# state machine using it.

https://github.com/adrian-thurston/ragel

[JimBobSquarePants]

First test passes in the state machine tests! Fingers crossed this is a sign of things to come.

[JimBobSquarePants]

General question, should we be commiting .generated. files or should we be generating them as part of the build?

Good question. It's definitely a manual process just now plus there's a bit of a chicken/egg situation going on with the UnicodeData class that references all the generated tries. I'm sure there's smarter ways to do it but I always end up regretting messing around with custom build config.