Hack icon indicating copy to clipboard operation
Hack copied to clipboard

Set SOURCE_DATE_EPOCH relative to git commit time to create reproducible timestamps/checksum values in the OT tables

Open chrissimpkins opened this issue 7 years ago • 2 comments

This will permit us to create reproducible OT table build structure at a given git commit. At the moment, build time is used to define the timestamp in the compiled font and by definition creates a time-dependent difference in the binary that is not associated with the source used for the compile process. This opens up the possibility to more easily use text diffs for comparison across build tooling versions, Python interpreter versions, different platforms, etc. as well as comparison of hashes of the font binary for differences.

Discussion with @anthrotype who suggested this approach begins at this post:

https://github.com/source-foundry/Hack/issues/398#issuecomment-367432102

A comparison of current builds across Py2 and Py3 interpreters (and venv isolated pip installed Python dependencies) yielded only these timestamps and checksum values calculated based upon them as differences in the compiled fonts:

https://github.com/source-foundry/Hack/issues/398#issuecomment-367427666

chrissimpkins avatar Feb 21 '18 19:02 chrissimpkins

With the latest release of the fontmake compiler and its ufo2ft dependency, we were able to get to fully reproducible font builds at every git commit using the git commit Unix timestamp of the checked out commit at build time.

We accomplished this by setting the SOURCE_DATE_EPOCH environment variable to the git commit Unix timestamp value with the following in the shell scripts:

export SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD)

and modifying all fontTools based post-compile writes to the binaries to turn off recalculation of timestamps on new file writes. This was necessary in the DSIG table write and the fstype fix scripts.

The modification was to TTFont object instantiation with the following:

font = ttLib.TTFont(file=path, recalcTimestamp=False)

Notably, the TTFA table write by ttfautohint does not appear to update the checksums/timestamps in the binary and no modifications were necessary there.

These changes are now in the dev branch and will be included as part of the v4.000 tagged release. You can verify the changes by checking out a git commit after 752fbdb76dd9148adfe7e00158bb40f5d59e9ab1 and comparing SHA1 checksum values across files that are built at different times for that commit. They should now be the same at a given git commit irrespective of the time that the commit is built.

👍 Big thanks to Cosimo, Khaled, and gang for the changes in the compilers that supported this. This is extremely helpful.

cc: @anthrotype @khaledhosny

chrissimpkins avatar Jun 13 '18 22:06 chrissimpkins

@paride @fabiangreffrath the changes mentioned in https://github.com/source-foundry/Hack/issues/401#issuecomment-397102332 will get you to Debian-specific reproducible/deterministic builds at every git commit. I can confirm that SHA1 checksums do not differ across different build times. You will need to support the fontmake and ufo2ft (fontmake dependency) versions as of fontmake v1.5 release to achieve this. This was a topic that Fabian raised a few months ago. I am hoping to push all of the build system changes (in v4.000) that we discussed over the last few months by late June/early July. Almost there...

chrissimpkins avatar Jun 14 '18 19:06 chrissimpkins