ttLib unparser should call GDEFUnparser

[marekjez86]

1. I took https://github.com/googlefonts/noto-source/tree/master/src/NotoSansBatak and build an OTF file https://github.com/googlefonts/noto-fonts/blob/master/unhinted/otf/NotoSansBatak/NotoSansBatak-Regular.otf
2. I ran otf2fea of the OTF file to extract new fea; I also modified the source to use this fea obtaining the sources contained in this zip file NotoSansBatak.zip
3. I build a new OTF using fontmake
4. I compared the new OTF with the previous OTF expect that they would be virtually the same, but they weren't (see the results of tools below)
5. the diffenator listed differences as follows (is this difference acceptable/expected in the new font)? :

gdef_base missing: 50

glyph
uni1BC1
uni1BC0
uni1BFD
uni1BD5
uni1BD7
uni1BC5
uni1BDB
uni1BFC
uni1BDC
uni1BC4
uni1BD9
uni1BC3
uni1BCF
uni1BDA
uni1BDF
uni1BCC
uni1BD4
uni1BC2
uni1BD0
uni1BE1
uni1BCB
uni1BC9
uni1BE3
uni1BC6
uni1BD3
uni1BD8
uni1BCA
uni1BD2
uni1BC8
uni1BCE
uni1BDE
uni1BE0
uni1BE5
uni1BE4
uni1BE2
uni1BFE
uni1BC7
uni1BD1
uni1BD6
uni1BCD
uni1BDD
uni1BFF
uni25CC
uni1BF2
uni1BEC
uni1BEA
uni1BF3
uni1BEB
uni1BE7
batak_null

gdef_mark new: 6

glyph
uni1BF2
uni1BEC
uni1BEA
uni1BF3
uni1BEB
uni1BE7

the following zip file contains the output of various font tools (including two diff ones comparing the original font with the new one) NotoSansBatak-20210223-1520.zip

[simoncozens]

OK, I understand this. The otf->fea process went through a few stages; first it generated feaLib output directly, then it generated a FontFeatures object as an intermediary and then used that to generate feaLib output. However, at that stage, FontFeatures did not support GDEF tables. Now it does, so we need to rewrite the GDEFUnparser to populate a FontFeatures object, and make sure that the GDEFUnparser is being called again.

[simoncozens]

Wait a minute, otf2fea does call the old GDEF unparser if you pass the --gdef flag. Try that.

[marekjez86]

I'll try that for all. I'll also follow up on DaveC suggestion to recompile all of the OTF files first with the current fontmake (some of the previous files were built about 2-4 months ago, so it's possible that it had fontmake issues still).

[marekjez86]

@simoncozens @davelab6 : I rebuilt OTF files per Dave's suggestion. I also ran otf2fea with --gdef flag per Simon's suggestion.

The file reported above now doesn't compile a font because: fontmake: Error: In 'NotoSansBatak.glyphs' -> 'master_ufo/NotoSansBatak-Regular.designspace' -> 'master_ufo/NotoSansBatak-Regular.ufo': Compiling UFO failed: <features>:352:5: Glyph uni1BE7 was assigned to a different class at <features>:347:5

Also about 25% of fontmake compiles failed; I also haven't checked the resulting font files

fontmake identified errors are as follows: Glyph * was assigned to a different class at

fontmake: Error: In 'Arimo-BoldItalic.glyphs' -> 'master_ufo/Arimo-BoldItalic.designspace' -> 'master_ufo/Arimo-BoldItalic.ufo': Compiling UFO failed: <features>:4346:5: Glyph commaaccent was assigned to a different class at <features>:4341:5
fontmake: Error: In 'Arimo-Italic.glyphs' -> 'master_ufo/Arimo-Italic.designspace' -> 'master_ufo/Arimo-Italic.ufo': Compiling UFO failed: <features>:4388:5: Glyph commaaccent was assigned to a different class at <features>:4383:5
fontmake: Error: In 'Arimo-Regular.glyphs' -> 'master_ufo/Arimo-Regular.designspace' -> 'master_ufo/Arimo-Regular.ufo': Compiling UFO failed: <features>:4449:5: Glyph commaaccent was assigned to a different class at <features>:4444:5
fontmake: Error: In 'Cousine-Italic.glyphs' -> 'master_ufo/Cousine-Italic.designspace' -> 'master_ufo/Cousine-Italic.ufo': Compiling UFO failed: <features>:550:5: Glyph hookabovecomb was assigned to a different class at <features>:545:5
fontmake: Error: In 'Cousine-Regular.glyphs' -> 'master_ufo/Cousine-Regular.designspace' -> 'master_ufo/Cousine-Regular.ufo': Compiling UFO failed: <features>:546:5: Glyph hookabovecomb was assigned to a different class at <features>:541:5
fontmake: Error: In 'Tinos-Bold.glyphs' -> 'master_ufo/Tinos-Bold.designspace' -> 'master_ufo/Tinos-Bold.ufo': Compiling UFO failed: <features>:3172:5: Glyph uni03080304.cap was assigned to a different class at <features>:3167:5
fontmake: Error: In 'Tinos-Italic.glyphs' -> 'master_ufo/Tinos-Italic.designspace' -> 'master_ufo/Tinos-Italic.ufo': Compiling UFO failed: <features>:3101:5: Glyph uni03080304.cap was assigned to a different class at <features>:3096:5
fontmake: Error: In 'Tinos-Regular.glyphs' -> 'master_ufo/Tinos-Regular.designspace' -> 'master_ufo/Tinos-Regular.ufo': Compiling UFO failed: <features>:3225:5: Glyph uni03080304.cap was assigned to a different class at <features>:3220:5
fontmake: Error: In 'NotoSansBatak.glyphs' -> 'master_ufo/NotoSansBatak-Regular.designspace' -> 'master_ufo/NotoSansBatak-Regular.ufo': Compiling UFO failed: <features>:352:5: Glyph uni1BE7 was assigned to a different class at <features>:347:5
fontmake: Error: In 'NotoSansGrantha.glyphs' -> 'master_ufo/NotoSansGrantha-Regular.designspace' -> 'master_ufo/NotoSansGrantha-Regular.ufo': Compiling UFO failed: <features>:16663:5: Glyph zerocomb_gran was assigned to a different class at <features>:16658:5
fontmake: Error: In 'NotoSansHanunoo.glyphs' -> 'master_ufo/NotoSansHanunoo-Regular.designspace' -> 'master_ufo/NotoSansHanunoo-Regular.ufo': Compiling UFO failed: <features>:80:5: Glyph uni1734 was assigned to a different class at <features>:75:5
fontmake: Error: In 'NotoSansLepcha.glyphs' -> 'master_ufo/NotoSansLepcha-Regular.designspace' -> 'master_ufo/NotoSansLepcha-Regular.ufo': Compiling UFO failed: <features>:361:5: Glyph uni1C29.wide was assigned to a different class at <features>:356:5
fontmake: Error: In 'NotoSansLimbu.glyphs' -> 'master_ufo/NotoSansLimbu-Regular.designspace' -> 'master_ufo/NotoSansLimbu-Regular.ufo': Compiling UFO failed: <features>:126:5: Glyph uni1923 was assigned to a different class at <features>:121:5
fontmake: Error: In 'NotoSansRejang.glyphs' -> 'master_ufo/NotoSansRejang-Regular.designspace' -> 'master_ufo/NotoSansRejang-Regular.ufo': Compiling UFO failed: <features>:103:5: Glyph uniA952 was assigned to a different class at <features>:98:5
fontmake: Error: In 'NotoSansSaurashtra.glyphs' -> 'master_ufo/NotoSansSaurashtra-Regular.designspace' -> 'master_ufo/NotoSansSaurashtra-Regular.ufo': Compiling UFO failed: <features>:146:5: Glyph uniA880 was assigned to a different class at <features>:141:5
fontmake: Error: In 'NotoSansSylotiNagri.glyphs' -> 'master_ufo/NotoSansSylotiNagri-Regular.designspace' -> 'master_ufo/NotoSansSylotiNagri-Regular.ufo': Compiling UFO failed: <features>:188:5: Glyph uniA823 was assigned to a different class at <features>:183:5
fontmake: Error: In 'NotoSerifBalinese.glyphs' -> 'master_ufo/NotoSerifBalinese-Regular.designspace' -> 'master_ufo/NotoSerifBalinese-Regular.ufo': Compiling UFO failed: <features>:704:5: Glyph uni1B04 was assigned to a different class at <features>:699:5
fontmake: Error: In 'NotoSerifGrantha.glyphs' -> 'master_ufo/NotoSerifGrantha-Regular.designspace' -> 'master_ufo/NotoSerifGrantha-Regular.ufo': Compiling UFO failed: <features>:16737:5: Glyph zerocomb_gran was assigned to a different class at <features>:16732:5

Within a named lookup block, all rules must be of the same lookup type and flag

fontmake: Error: In 'NotoSansSogdian.glyphs' -> 'master_ufo/NotoSansSogdian-Regular.designspace' -> 'master_ufo/NotoSansSogdian-Regular.ufo': Compiling UFO failed: <features>:55:5: Within a named lookup block, all rules must be of the same lookup type and flag

[marekjez86]

NotoSansHanunoo sources NotoSansHanunoo.zip

[marekjez86]

NotoSansSogdian sources NotoSansSogdian.zip

[marekjez86]

@davelab6 : Stats now out of 64 easy sources with --gdef

• 27 failed to compile for various reasons
• some of 15 compilations that succeeded might be OK (so it means that we might have some valid conversions among them)
• 22 compilations generated different fonts that the original OTF (need more investigation)

[simoncozens]

OK, I think the "glyph was assigned a different class" might be a fontmake problem. I suspect it is adding its own GDEF to the GDEF in the features.fea file. What I'd like you to try for those files - just as a test - is:

• Run otf2fea --gdef Whatever.otf on the built font.
• Run fontmake -g Whatever.glyphs -o otf on the original Glyphs source (i.e. with no features.fea prefix)
• run fonttools feaLib -o master_otf/Whatever.otf features.fea master_otf/Whatever.otf
• Then compare the original (Whatever.otf) with the new fontmake-plus-feaLib version master_otf/Whatever.otf.

[simoncozens]

Yes, that's exactly what's happening. fontmake is adding its own GDEF. And it's getting it wrong...

[simoncozens]

See googlefonts/fontmake#544, googlefonts/fontmake#702.

[simoncozens]

Reflecting some more on this: the fontmake answer would be "fontmake is getting GDEF wrong because your Glyphs sources are wrong". And there's some merit in that. U+1BE7 looks very much like a mark but we haven't said that we want it to be a base, even though (looking at the Noto binary) we do want it to be a mark.

But the fontmake answer would continue "You need to fix your sources. If you want U+1BE7 to be a base instead of a mark, then you need to set it as a base in the Glyphs source by going to the Glyph Info window and setting Category to Letter, otherwise you will get the auto-guessed default."

Which is certainly one answer.

But my answer would be, "No, I am the font engineer, and sometimes designers send over bad sources, and I need to deal with that situation. And the mechanism I have for dealing with that is by putting my own GDEF definitions in the feature file, which is the bit of the process under my control. So when I do put my own definitions in the feature file, fontmake should actually use them."

In other words, I think the idea in googlefonts/fontmake#709 for adding a flag to stop fontmake generating its own GDEF tables is a good one, and will solve your problem if it gets merged.

[marekjez86]

@simoncozens : thank you very much.. Feels like we made a real good progress on this, but a real solution might be to have someone (not me, I don't qualify :-) ) working on this conversion; someone who understands the feature files, sources, who is able to look at fontmake and diffenator to see if what they do makes sense (or even if they act correctly) to make further progress on the conversion project.

[davelab6]

I wonder if @behdad might be interested to make otf2fea work to 100%.

[davelab6]

@simoncozens it might help Behdad to assess how feasible this is and how much effort, if you could post a issue (or set of issues with a milestone/tag) for what know to be left to do, so he can take a look

[davelab6]

Per chat with Marek and Behdad about this today, I see there are 2 milestones:

1st, "80/20" (which I think maybe you already passed) such that this is a helpful tool to accerlate "by hand" conversion of MTI/binary fonts to fresh FEA (or, better imho, FEE) syntax, doing the easy 80% of the work (which for some fonts is 100%) and leaving the "hard" work to be done by hand (which for some fonts is also 100%, haha).

2nd, "100%", where it reliably parses any representation of any OT lookups and can write them back out in any syntax. This seems theoretically possible to me (and Behdad concurred) but maybe difficult to implement, such that only Behdad or Simon with dedicated time and space to do it, can do it.

[simoncozens]

Well, there are two issues here; we haven't actually hit any otf2fea issues yet, just fontmake ones (and possibly fontdiffenator ones!).

Right now you should be able to use fontmake -g on the unmodified glyphs source, and use fonttools feaLib features.fea Foo.ttf to stuff the feature tables in there correctly, and build a font. If @behdad or I can get the above fontmake PR rebased and merged, we should be able to make fonts using fontmake in the standard way. That's the 80/20%.

The remaining challenge (I'd say it was less than 20%) is to deal with fonts with mark-to-ligature rules, which are vanishingly rare but annoying; see #38.

[anthrotype]

the above fontmake PR ...

@simoncozens FYI https://github.com/googlefonts/fontmake/pull/709 is now merged (released in v2.3.1), I hope that unblocks you.

[simoncozens]

Thanks! @marekjez86, please upgrade your fontmake, use otf2fea --gdef Whatever.otf, and fontmake -g Whatever.glyphs --no-generate-GDEF.

[marekjez86]

Will do. Thanks for making this happen.

On Mon, Mar 1, 2021 at 8:23 AM Simon Cozens [email protected] wrote:

Thanks! @marekjez86 https://github.com/marekjez86, please upgrade your fontmake, use otf2fea --gdef Whatever.otf, and fontmake -g Whatever.glyphs --no-generate-GDEF.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/simoncozens/fontFeatures/issues/35#issuecomment-788080309, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADFUS7ZXL2ETNQ5UXEDSUVLTBO5QJANCNFSM4YDVYJKQ .

[davelab6]

@marekjez86 can this now be closed? :)