Hack
Hack copied to clipboard
source-foundry
Pin Python build dependencies to defined versions
Establish builds that can be built in a reproducible fashion at a specific commit at any point in time.
Suggestion by @anthrotype to create requirements.txt file:
https://github.com/source-foundry/Hack/issues/398#issuecomment-367435437
Modified following discussions to use pipenv with Pipfile + Pipfile.lock Python build dependency pinning.
TODO:
- [x] create pipenv venv with new Pipfile and Pipfile.lock files to define pinned Python build dependency versions https://github.com/source-foundry/Hack/commit/84900249b9b3a67beebb888e1e24c84fbfc4fd63
- [x] modify build scripting to use the pipenv venv to build font files under Python 3 with pinned build dependency versions as defined in the Pipfile.lock file
- [x] modify ttf build scripts
- [x] modify woff build scripts
If you don’t mind the noise in your inbox pyup.io runs a bit that will automatically send PRs to update a requirements.txt files in your repo whenever any package mentioned there has a new version published in PyPI.
Good service. It serves the changelogs as part of the PR too when they are available through its automated changelog parsing method. Good idea. Thanks Cosimo!
Yeah, it’s good. You may want to configure it’s frequency to say once a week or so because it gets pretty quickly annoying. Check the configuration file in fontmake repo for example
Convert to the following approach in build scripting to install all Python dependencies:
pip install -r /path/to/requirements.txt
and eliminate all pip install [project] commands in the scripts. This means that we should be documenting the best approach for builds to be within a virtualenv.
@anthrotype I could use your feedback about how to accomplish this on my development machine. Do you suggest that I create a virtualenv, activate & install requirements.txt dependencies in the venv, build from this, delete the venv after build, and create new venv for every new build? I've used venv for one off isolation of dependencies but have never kept one around for ongoing development as other things change on the machine. Homebrew versions of the Python interpreter will be rolling on my development system and this is desired for other reasons on that machine. Possible to swap out the Python 3 interpreter in an existing venv so that the libraries and executables do not need to be reinstalled each time this changes? Is the Python interpreter completely isolated in the venv or does it break if the venv is defined with Homebrew installed Python where cleanups remove previous system installed versions of Pythons?
Possible to swap out the Python 3 interpreter in an existing venv so that the libraries and executables do not need to be reinstalled each time this changes?
no, each venv is connected to one specific python interpreter, cannot be shared. You need to create two different ones.
Is the Python interpreter completely isolated in the venv and does it break if the venv is defined with Homebrew installed Python where cleanups remove previous versions of the installed Pythons?
the venv will break if you update the host python, yeah. Venvs are meant to be throw-away and easy to delete and recreate, the only things that defines then is the requirements.txt with all the dependencies pinned to a specific version.
Probably best to script the venv creation then. OK, that is very helpful. Thanks Cosimo.
there's a new kid in town, called pipenv, which has recently also become officially blessed by the Python Packaging Authority (PyPA), though I haven't checked it out yet. It seems more geared towards "applications" than "libraries" (which is mostly what I deal with). Perhaps it may work in your case though. Try it out. There's also a tutorial of PyPUG https://packaging.python.org/tutorials/managing-dependencies/?highlight=pipenv
Interesting. It looks like the Python scripts are run like this for the local install of Python isolated through pipenv:
pipenv run python main.py
or maybe this to use Python3?:
pipenv run python3 main.py
I wonder how you use it to run isolated executables installed with that approach. Will take a deeper dive into the docs for both that and virtualenv to see what approach will work the best. Thanks again Cosimo.
By the way, this was a very good suggestion for the project. Thanks for the push to get this moving. It needs to happen. We are going to pin all build dependency versions across the Python projects and non-Python projects in the tooling as well as eliminate pathways to build from system installed versions of build tools when detected as present on the build machine. Work is underway.
Right now we have those versions pinned but will default to a system installed version of ttfautohint when present. This is going to be eliminated
As a down the road change, we may transition to fontTools for the web font builds. I would like to eliminate the subsetting approach here (very much a hack of fontmake's subsetting capabilities) and use pyftsubset in the future. We are going to make multiple subset files and the capacity to easily define those in pyftsubset is a better approach for us. It was going to be part of the next release but I bumped this back a bit. Probably part of our next release work.
Harfbuzz and Freetype only for ttfautohint, right?
basically, all the stuff you mention can be done one way or another with some python module, and all the build requirements can be easily controlled with pip.
FontTools can compress woff (optionally with zopfli) and woff2 as well.
Harfbuzz and Freetype only for ttfautohint, right?
Yep. Only for ttfautohint
basically, all the stuff you mention can be done one way or another with some python module, and all the build requirements can be easily controlled with pip.
Yes that is what I am thinking. I opened an issue report to consider transition to your ttfautohint-py tool when you mentioned it to me. This may be a good way to control these versions.
FontTools can compress woff (optionally with zopfli) and woff2 as well.
See https://github.com/source-foundry/Hack/issues/400#issuecomment-368601202. I believe that you have to install the Python bindings for zopfli and brotli to compile zopfli compressed woff and woff2 (brotli by default for the latter) correct? I have these installed locally and it will not be a problem for our builds in this repo but I will break downstream Linux distro package builds if those bindings are not available and we make a transition in the default build scripting. It would be worth making them available on these distros if they are not and this is something that we can discuss with those teams as the work on this begins. There are definite plans to transition to pyftsubset in the future. The build from source issues on Linux distros are new and I am still trying to figure out how to navigate responsibilities as an upstream maintainer...
good, let me know if I can help, as I maintain both of these bindings.
-
py-zopfli https://github.com/obp/py-zopfli https://pypi.python.org/pypi/zopfli
-
brotli (the upstream repo for the C library also contains a python extension module) https://github.com/google/brotli https://pypi.python.org/pypi/Brotli
Alternatively this other brotli binding also works (has a compatible interface with the upstream brotli package, and is built with cffi so supposedly is more compatible with pypy)
- https://pypi.python.org/pypi/brotlipy
- https://github.com/python-hyper/brotlipy/
As a related aside, if ttfautohint-py (edit: and the brotli/zopfli bindings) are not currently on the radar for Linux distro packagers, it would be worth making sure that someone on the distros that build fonts from source (Debian/Ubuntu + Fedora in our experience here) are aware of this and begin work to go through the vetting process. The process seems to take some time and is a consideration for typeface projects that want to release with your software on these Linux distros.
I've tried to figure out if there are distro specific working groups that determine what tooling is worth packaging for font builds. For Debian there is a fonts group and they maintain GitLab repos for work on their approved packages. I believe that Shawn Starr mentioned to me that there is something along these lines on Fedora but as I recall it was not as formally defined? In any case, it would be helpful for tool developers to have representation on these committees, particularly if you are a user of one of the distros. We (typeface development community) should help to define the important and needed tools there IMO. We are trying to help through this project and this may be an avenue for ttfautohint-py if it is not currently packaged. Unfortunately under free software guidelines of certain distros there are significant constraints on what decisions we can make upstream if we want to release there.
Would you be willing to pitch in on the work to transition the current tooling to those projects? Knowing how many projects you are involved in as a result of watching many of your repositories this will be a silly question but do you happen to have any extra time in the next few months?
I am more than happy to approach our Fedora and Debian packagers to see if they would be willing to begin looking into these new dependencies for us.
do you happen to have any extra time in the next few months?
I'd love to, but don't have any spare time at the moment, sorry :(
This has the potential to (and likely will) break source builds since I plan to make this a default and not optional. We are going to provide new make targets for those who want to build with rolling system installed versions as in the current approach for Python build tooling but this will no longer be the default make-based build approach supported here.
I am going to push our current dev branch design/etc changes as v3.003 and will include the changes that you suggested here and in some of the other open Python tooling related threads as part of a new major version release shortly thereafter. v4.000 will not include a transition to pyftsubset or ttfautohint-py as part of the default build process. Let's give packagers a bit of time to work through approval processes for the other projects before we add them to the default build. Of the two new dependencies, I would like to create our web font subsets sooner rather than later so I may use an approach outside of the default build process (in a non-build breaking fashion) to generate these files until we formally transition as part of the default build defined here. That change will represent another new major version (likely v5.000). We have a good approach to pin ttfautohint and its dependencies at the moment so there is no urgency to make the transition to ttfautohint-py short of elimination of all of the C/C++ source compiles right now.
Have been reading on pipenv and this may be the approach that we take here. Lots to like. Automation and standardization of the venv location, easy to make, easy to remove, simple definitions of pinned versions, simple upgrade of dependency versions (including the Python version) in the venv, pip security features... list goes on.
Rather than generate a requirements.txt, this will involve generating Pipfile and Pipfile.lock files in the repository root.
General recommendations from Kenneth Reitz's documentation:
- Generally, keep both Pipfile and Pipfile.lock in version control.
- Do not keep Pipfile.lock in version control if multiple versions of Python are being targeted.
- Specify your target Python version in your Pipfile’s [requires] section. Ideally, you should only have one target Python version, as this is a deployment tool.
New dependency installs for builders (including CI testing services)
pipenv + pyenv. The latter supports Python interpreter installs of versions that are not defined on the build system, automated with pipenv if the locked interpreter version is not detected.
$ pip install pipenv
$ curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash
Initial install of Hack build dependencies to define versions
Creates Pipfile in repository root, defines Python interpreter version to be used, and creates a new venv with defined build dependencies at current PyPI release versions
$ pipenv install --python 3.6 fontTools fontmake
View graph of deps
$ pipenv graph
Lock dependency versions for reproducible builds with Python dependencies
Creates Pipfile.lock file in repository root with pinned dependency versions
$ pipenv lock
Scripted install of version pinned Hack build dependencies once Pipfile.lock write takes place
$ pipenv install --ignore-pipfile fontTools fontmake
Check for outdated build dependencies
$ pipenv update --outdated --dry-run
Upgrade build dependencies for testing before locking as defined build versions
$ pipenv install --skip-lock --python [Py version] fontTools fontmake
# build + test
$ pipenv lock
Remove packages that are no longer needed as project dependencies, then re-lock
$ pipenv uninstall --lock [package]
Use of venv versions of build dependencies in shell scripts
# font compile
pipenv run fontmake [args]
# post compile modifications
pipenv run python [Python script] [args]
Dependency safety checks
$ pipenv check
Remove the venv
$ pipenv --rm
Anyone used pipenv in production settings / have feedback on this?
@anthrotype it is possible to create a requirements.txt file with pipenv so that pyup could be used for automated push/CI testing against updated build dependencies. Wondering whether we want to have both a requirements.txt file and Pipfile 's in the root in order to support this?
$ pipenv install --python 3.6 fontTools fontmake
Creating a virtualenv for this project…
Using /usr/local/bin/python3.6m to create virtualenv…
⠋Running virtualenv with interpreter /usr/local/bin/python3.6m
Using base prefix '/usr/local/Cellar/python/3.6.4_3/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/ces/.local/share/virtualenvs/Hack-ZEQHr9Bh/bin/python3.6
Also creating executable in /Users/ces/.local/share/virtualenvs/Hack-ZEQHr9Bh/bin/python
Installing setuptools, pip, wheel...done.
Virtualenv location: /Users/ces/.local/share/virtualenvs/Hack-ZEQHr9Bh
Creating a Pipfile for this project…
Installing fontTools…
Collecting fontTools
Using cached fonttools-3.24.1-py2.py3-none-any.whl
Installing collected packages: fontTools
Successfully installed fontTools-3.24.1
Adding fontTools to Pipfile's [packages]…
Installing fontmake…
Collecting fontmake
Using cached fontmake-1.4.0-py2.py3-none-any.whl
Collecting ufo2ft>=1.1.0 (from fontmake)
Downloading ufo2ft-1.1.0-py2.py3-none-any.whl (47kB)
Collecting booleanOperations>=0.8.0 (from fontmake)
Downloading booleanOperations-0.8.0-py2.py3-none-any.whl
Requirement already satisfied: fonttools>=3.21.2 in /Users/ces/.local/share/virtualenvs/Hack-ZEQHr9Bh/lib/python3.6/site-packages (from fontmake)
Collecting cu2qu>=1.3.0 (from fontmake)
Downloading cu2qu-1.4.0-py2.py3-none-any.whl
Collecting defcon>=0.3.5 (from fontmake)
Downloading defcon-0.3.5-py2.py3-none-any.whl (209kB)
Collecting MutatorMath>=2.1.0 (from fontmake)
Downloading MutatorMath-2.1.0-py2.py3-none-any.whl
Collecting glyphsLib>=2.2.1 (from fontmake)
Downloading glyphsLib-2.2.1-py2.py3-none-any.whl (272kB)
Collecting ufoLib>=2.1.0 (from ufo2ft>=1.1.0->fontmake)
Downloading ufoLib-2.1.1-py2.py3-none-any.whl (93kB)
Collecting compreffor>=0.4.5 (from ufo2ft>=1.1.0->fontmake)
Downloading compreffor-0.4.6-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (118kB)
Collecting pyclipper>=1.0.5 (from booleanOperations>=0.8.0->fontmake)
Downloading pyclipper-1.1.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (279kB)
Collecting fontMath>=0.4.4 (from MutatorMath>=2.1.0->fontmake)
Downloading fontMath-0.4.4-py2.py3-none-any.whl
Installing collected packages: ufoLib, defcon, pyclipper, booleanOperations, cu2qu, compreffor, ufo2ft, fontMath, MutatorMath, glyphsLib, fontmake
Successfully installed MutatorMath-2.1.0 booleanOperations-0.8.0 compreffor-0.4.6 cu2qu-1.4.0 defcon-0.3.5 fontMath-0.4.4 fontmake-1.4.0 glyphsLib-2.2.1 pyclipper-1.1.0 ufo2ft-1.1.0 ufoLib-2.1.1
Adding fontmake to Pipfile's [packages]…
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (d9bb77)!
Installing dependencies from Pipfile.lock (d9bb77)…
🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 12/12 — 00:00:01
To activate this project's virtualenv, run the following:
$ pipenv shell
This command automatically makes the Pipfile.lock file. No need for a pipenv lock step at this stage.
Generated Pipfile:
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"
[packages]
fonttools = "*"
fontmake = "*"
[dev-packages]
[requires]
python_version = "3.6"
This file locks the Python interpreter version. Pipfile.lock locks the package versions. OK to leave this file specified as any version of fontmake and fonttools.
pipenv graph shows the following:
fontmake==1.4.0
- booleanOperations [required: >=0.8.0, installed: 0.8.0]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- pyclipper [required: >=1.0.5, installed: 1.1.0]
- ufoLib [required: >=2.0.0, installed: 2.1.1]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- cu2qu [required: >=1.3.0, installed: 1.4.0]
- fonttools [required: >=3.18.0, installed: 3.24.1]
- ufoLib [required: >=2.1.1, installed: 2.1.1]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- defcon [required: >=0.3.5, installed: 0.3.5]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- ufoLib [required: >=2.1.0, installed: 2.1.1]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- fonttools [required: >=3.21.2, installed: 3.24.1]
- glyphsLib [required: >=2.2.1, installed: 2.2.1]
- defcon [required: >=0.3.0, installed: 0.3.5]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- ufoLib [required: >=2.1.0, installed: 2.1.1]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- fonttools [required: >=3.4.0, installed: 3.24.1]
- MutatorMath [required: >=2.0.4, installed: 2.1.0]
- defcon [required: >=0.3.5, installed: 0.3.5]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- ufoLib [required: >=2.1.0, installed: 2.1.1]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- fontMath [required: >=0.4.4, installed: 0.4.4]
- fonttools [required: >=3.3.0, installed: 3.24.1]
- ufoLib [required: >=2.0.0, installed: 2.1.1]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- ufoLib [required: >=2.0.0, installed: 2.1.1]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- MutatorMath [required: >=2.1.0, installed: 2.1.0]
- defcon [required: >=0.3.5, installed: 0.3.5]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- ufoLib [required: >=2.1.0, installed: 2.1.1]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- fontMath [required: >=0.4.4, installed: 0.4.4]
- fonttools [required: >=3.3.0, installed: 3.24.1]
- ufoLib [required: >=2.0.0, installed: 2.1.1]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- ufoLib [required: >=2.0.0, installed: 2.1.1]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- ufo2ft [required: >=1.1.0, installed: 1.1.0]
- booleanOperations [required: >=0.7.1, installed: 0.8.0]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- pyclipper [required: >=1.0.5, installed: 1.1.0]
- ufoLib [required: >=2.0.0, installed: 2.1.1]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- compreffor [required: >=0.4.5, installed: 0.4.6]
- fonttools [required: >=3.1, installed: 3.24.1]
- cu2qu [required: >=1.2.0, installed: 1.4.0]
- fonttools [required: >=3.18.0, installed: 3.24.1]
- ufoLib [required: >=2.1.1, installed: 2.1.1]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- defcon [required: >=0.3.4, installed: 0.3.5]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- ufoLib [required: >=2.1.0, installed: 2.1.1]
- fonttools [required: >=3.1.2, installed: 3.24.1]
- fonttools [required: >=3.17.0, installed: 3.24.1]
- ufoLib [required: >=2.1.0, installed: 2.1.1]
- fonttools [required: >=3.1.2, installed: 3.24.1]