googlefonts
New builder
New builder sketch
The current builder does a good job of taking source files and producing a range of outputs suitable for Google Fonts use. However, it statically encodes the process for turning a given source (Glyphs or Designspace) into a set of output files. We can turn on and off some of those outputs, and we can (minimally) customize the process, but we don't have the ability to introduce new "steps" or transformations. And yet we do have a number of font production and post-production processes which can't be currently achieved with the existing builder:
- A different set of outputs for Noto, requiring its own
notobuilder. - Noto also requires adding Latin subsets (and other non-Latin fonts will require adding a subset of Latin Kernel); this is a source-source (UFO-level) transformation.
- Adding COLRv1 paints using paintcompiler.
- Noto's UI fonts require cmap remapping, baseline shifts and transformations.
- Using hb-subset to slim down / repack the font.
- Handwriting and other "mega-fonts" will require variable font subspacing, feature freezing and more.
- Other users have wanted PS hints and other custom operations.
Currently this requires horrible Makefile hacks or ad-hockery. Rod has also expressed a desire to have a single build process that Just Works to build everything we onboard - and I think it can be done...
To see how to do it, we need to take a step back before we step forwards. I started out by thinking about how an end-user would represent, in a config file, the desired set of build operations for a custom production process. Essentially we are representing a tree structure where different sources are transformed, saved, passed to other operations, saved as a different file, and so on. But the easiest way to represent that tree structure is just to list the operations and let the computer work out the most efficient way to do them. For example, let's compile NotoSerifEthiopic.glyphs into three variable font targets: an unhinted font, a slimmed-down VF for Android, and a hinted VF. We could somehow try to represent "compile the VF, save as unhinted; pass one copy to varLib.subset and save as slim-vf; pass another copy to ttfautohint and save as hinted". But it's actually easier to represent that as a set of distinct operations:
recipe:
fonts/NotoSansEthiopic/unhinted/variable-ttf/NotoSansEthiopic[wdth,wght].ttf:
- source: NotoSerifEthiopic.glyphs
- operation: buildVariable
- operation: buildStat
- operation: fix
fonts/NotoSansEthiopic/unhinted/slim-vf/NotoSansEthiopic[wdth,wght].ttf:
- source: NotoSerifEthiopic.glyphs
- operation: buildVariable
- operation: buildStat
- operation: subspace
- operation: hbsubset
fonts/NotoSansEthiopic/hinted/NotoSansEthiopic[wdth,wght].ttf:
- source: NotoSerifEthiopic.glyphs
- operation: buildVariable
- operation: buildStat
- operation: fix
- operation: autohint
and let the computer reconstruct the tree itself.
It turns out to be pretty easy to make the most efficient sequence for this: for each target, walk through the list of operations build a graph from a "source file" node to another "source file" or "binary file" node using the operation as the edge, labeling the final target node with the target's filename. So for target 1:
[NotoSerifEthiopic.glyphs]
|
| buildVariable
v
[ Binary File Node ]
|
| buildStat
v
[ Binary File Node ]
|
| fix
v
[unhinted/variable-ttf/...]
Then process target 2:
NotoSerifEthiopic.glyphs
|
| buildVariable
v
[ Binary File Node ]
|
| buildStat
v
[ Binary File Node ]
/ \
fix / \ subspace
/ \
[unhinted...] [ Binary File Node ]
|
| hbsubset
|
[ slim-vf/... ]
And so on; finally, label the unnamed file nodes with a temporary filename.
This graph structure can be very naturally transformed into a ninja build file, (let's say each "operation" such as buildVariable is defined by a Python module which ensures that the incoming file and outgoing file are compatible and writes the ninja rules for that transformation) and then we're basically done.
In the configuration YAML, as well as operation we could specify additional arguments passed to the Python modules which implement each operation; we could even have an escape-hatch exec step which calls an arbitrary binary (although we would want to provide modules for most operations that people might want to do on a font):
fonts/NotoSansMath/hinted/NotoSansMath-Regular.ttf:
- source: NotoSerifEthiopic.glyphs
- operation: buildTTF
- operation: exec
cmd: python3 scripts/add-math-table.py -o $out $in
- operation: autohint
Stepping forward
If we do that, we have a very flexible builder, but we've gone backwards in the sense that we're now requiring the user to specify all the steps and all the outputs, when the current builder works out what we need in terms of the desired GF fonts artefacts and directory layouts. Ideally we'd like the new builder config to look like the old config in most cases - the user just provides the names of the source files, and we do the rest in the usual GF-y way.
To fix that, let's have the new builder read in the current source file, examine the sources and add its own recipe field to the in-memory data structure. Any additional entries provided by the user in the recipe field are taken as overrides. We could assume a default value in the config of recipe_provider: googlefonts to use the fontbuilder.recipes.googlefonts module to perform this examination/infilling, allowing for other recipe providers (noto, etc.)
Todos
- [x] Write a ufomerge operation to duplicate the functionality of notobuilder's subsets
- [x] Improve the handling of STAT table generation (only do it once with all VFs, not once per VF)
- [x] Write a cmap remapping / glyph swap operation
- [x] Make sure we can do everything that notobuilder.makeuivf currently does
- [x] Test, test, test
I'm loving the operations part. Having individual pieces which are still super reusable is a major win. I've only skimmed but I assume we can use this new builder as a hot fixer by skipping the fontmake part completely?
Yes, I think so. The only fiddly bit is that if you want to hotfix a file in-place, you need to say postprocess: fix instead of operation: fix. This is because with
recipe:
Foo.ttf:
- source: Foo.ttf
- operation: fix
the target of the ninja rule will be Foo.ttf which clearly already exists so ninja will do nothing. If you say
recipe:
Foo.ttf:
- source: Foo.ttf
- postprocess: fix
then the target of the ninja rule will be a "stamp file" and it'll do the right thing.
Recent builder1 changes to port:
- [x] otfautohint
- [x] WOFF after VTT
- [x] fix_hinted_font after VTT
- [x] removeOutlineOverlaps
- [x] reverseOutlineDirection
Hey Simon.
I've started reviewing this PR today.
Before I start delving into the code, I've decided to start testing it on existing families. Maven Pro currently fails with the traceback:
Traceback (most recent call last):
File "/Users/marcfoley/Type/tools/venv/bin/gftools", line 8, in <module>
sys.exit(main())
^^^^^^
File "/Users/marcfoley/Type/tools/Lib/gftools/scripts/__init__.py", line 91, in main
mod.main(args[2:])
File "/Users/marcfoley/Type/tools/Lib/gftools/builder/__init__.py", line 338, in main
pd = GFBuilder(args.config)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/marcfoley/Type/tools/Lib/gftools/builder/__init__.py", line 57, in __init__
automatic_recipe = self.call_recipe_provider()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/marcfoley/Type/tools/Lib/gftools/builder/__init__.py", line 88, in call_recipe_provider
return provider(self.config, self).write_recipe()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/marcfoley/Type/tools/Lib/gftools/builder/recipeproviders/googlefonts.py", line 62, in write_recipe
self.config["stat"].revalidate(stat_schema)
File "/Users/marcfoley/Type/tools/venv/lib/python3.11/site-packages/strictyaml/representation.py", line 109, in revalidate
result = schema(self._chunk)
^^^^^^^^^^^^^^^^^^^
File "/Users/marcfoley/Type/tools/venv/lib/python3.11/site-packages/strictyaml/validators.py", line 17, in __call__
self.validate(chunk)
File "/Users/marcfoley/Type/tools/venv/lib/python3.11/site-packages/strictyaml/compound.py", line 246, in validate
for item in chunk.expect_sequence():
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/marcfoley/Type/tools/venv/lib/python3.11/site-packages/strictyaml/yamllocation.py", line 105, in expect_sequence
self.expecting_but_found(expecting, "found {0}".format(self.found()))
File "/Users/marcfoley/Type/tools/venv/lib/python3.11/site-packages/strictyaml/yamllocation.py", line 44, in expecting_but_found
raise YAMLValidationError(
strictyaml.exceptions.YAMLValidationError: when expecting a sequence
in "<unicode string>", line 7, column 1:
stat:
^ (line: 7)
found a mapping
in "<unicode string>", line 25, column 1:
value: '900'
^ (line: 25)
Seems to be having issues with the STAT table in the config file.
I'll keep posting family issues until they all clear. I'll then take a look at the code.
@bramstein This PR is a major overhaul of the existing builder. The aim of this PR is to consolidate all of our custom build chains into one, since we have many. In order to do this, Simon has created a more granular approach to building fonts aka recipes + operations. It shouldn't affect any of your font builds since it is backwards compatible. I'd love you to test this PR just to make sure.
@simoncozens feel free to correct what I've written above or explain more.
@m4rc1e Thanks for letting us know. We'll give it a try (currently upgrading to the latest gftools), and then we can test.
Perhaps one observations, which might or might not be relevant here, is that cleanUp does not seem to cover it all. In particular, /instance_ttf and *.designspace* are left behind, which was not happening before.
Something is wrong with your venv. The builder2 branch has this:
https://github.com/googlefonts/gftools/blob/ed030d9bb04bc19ed761c114a9d949d6d66dd450/Lib/gftools/util/google_fonts.py#L43