Termite is a generic distributed compilation system.

The master distributes the compilation to workers. Workers run arbitrary binaries in a containerized FUSE mirror of the master's file system, and then ship the results back to the master.

CAVEATS

Work in progress.

COMPILE/INSTALL

• Install go.

• Install prerequisites:

  go install code.google.com/p/go.crypto/ssh go install github.com/hanwen/go-fuse/fuse

• Compiling:

  git clone https://github.com/hanwen/termite mkdir go ; cd go export GOPATH=$(pwd) (cd bin/mkbox ; make ) for d in bin/coordinator bin/worker bin/master bin/shell-wrapper do go install github.com/hanwen/termite/$d done sudo cp termite-make /usr/local/bin/ sudo cp bin/mkbox/mkbox /usr/local/bin/termite-mkbox sudo cp /tmp/go/bin/* /usr/local/bin/

• Make needs to be patched to use termite's shell wrapper:

  Add MAKE_SHELL variable to make.

  wget http://ftp.gnu.org/gnu/make/make-3.82.tar.bz2 tar xjf make-3.28.tar.bz2 cd make-3.82 && patch -p1 < ../termite/patches/make-*patch ./configure && make && make install

• Coreutils before 8.0 has buggy directory traversal, making 'rm -rf' flaky.

• Set resource limits: add the following to your /etc/security/limits.conf

  root soft nofile 5000 root hard nofile 5000

  • soft nofile 5000
  • hard nofile 5000

• Mount the source/object directories so termite can write xattrs, and noatime for performance improvements:

  mount -o remount,user_xattr,noatime my/device my/mountpoint

OVERVIEW

There are 5 binaries:

• Mkbox: a wrapper that sets up the containerization. Based on Brian Swetland's https://github.com/swetland/mkbox

• Coordinator: a simple server that administers a list of live workers. Workers periodically contact the coordinator.

• Worker: should run as root, and typically runs on multiple machines.

• Master: the daemon that runs on the machine. It contacts the coordinator to get a list of workers, and reserves job slots on the workers. Run it in the root of the writable directory for the compile. It creates a .termite-socket that the wrapper below uses.

• Shell-wrapper: a wrapper to use with make's SHELL variable.

The choice between remote and local can be set through the file .termite-localrc in the same dir as .termite-socket. The file is in json format, and you can find examples in the patches/ subdirectory. The default

[{ "Regexp": ".termite-make", "Local": true, "Recurse": true, "SkipRefresh": true }, { "Regexp": ".", "Local": false }]

(ie., only recursive make calls are run locally) should work for most projects, but for performance reasons, you might want to run more commands locally.

Typically, build-system commands should run locally (eg. make, cmake).

Commands that modify build artefacts should not run locally: local commands do not run inside a FUSE sandbox, so termite can't tell what files they modify, and how to update filesystem caches on the workers. By default, after executing a local command, the termite master scans for changed files. If you know this is not the case, you can skip this with SkipRefresh: true.

RUNNING

ssh-keygen -t rsa -b 1024 -f termite_rsa ${TERMITE_DIR}/bin/coordinator/coordinator -secret termite_rsa & ${TERMITE_DIR}/bin/worker/worker -jobs 4 -coordinator localhost:1233
-secret termite_rsa

cd ${PROJECT} ${TERMITE_DIR}/bin/master/master -jobs 4
-secret termite_rsa & termite-make -j20

PERFORMANCE

See below. The overhead of running in FUSE is 50 to 100%

SECURITY

• The worker runs binaries inside a containerized mount of a FUSE file system.

• Worker and master use plaintext TCP/IP, and use a shared secret with HMAC-SHA1 to authenticate the connection. See https://github.com/hanwen/termite/blob/master/termite/connection.go for details.

• Worker and master must trust each other, for the following reasons:

  • workers can request all publicly readable files from the master.

  • workers can cause the master to run arbitrary binaries as the user compiling.

  • the master can make the worker run arbitrary binaries as 'nobody'.

• The master will never serve files that have no group/other permissions.

• Wrapper and master run as the same user and use IPC unix domain sockets to communicate. The socket mode is 0700.

CAVEATS

• Hardlinks on the workers are translated to copies on the master.

TODO (by decreasing priority)

• Worker -> worker fetch
• Connection scheme: exp/ssh, security review?

SUCCESSFUL COMPILES

Termite timings by running master and single worker on the same machine. The smaller the package, the larger the overhead.

• coreutils 8.12 (1.6x slower, Lenovo T60, 2-core, make -j2)

• Make 3.82 (1.85x slower, Lenovo T60, 2-core)

• LLVM 2.9 (1.5 slower, Dell T5300 6-core, make -j12)

• GUILE 2.0.

• Must run inside srcdir.
• 1.1x slower, Dell T5300 6-core, make -j6

• Emacs 24

• 1.8x slower (Lenovo T60, make -j2)
• Must run in srcdir.

• Android Gingerbread.

DISCLAIMER

This is not an official Google product.