compiler-benchmark

Benchmarks compilation speeds of different combinations of languages and compilers. Supported languages are:

Languages with Natives Compilers

• C (using gcc, clang, cproc, and tcc),
• C++ (using g++ and clang++),
• D (using dmd ldmd2, and gdc),
• Go (using go or gccgo),
• Swift (using swiftc),
• Rust (using rustc),
• Nim (using nim),
• Julia (using julia),
• Ada (using gnatgcc),
• Zig (using zig), and
• V (using v),
• Vox (using vox),
• C3 (using c3c),
• Pareas (using pareas),

Languages with Bytecode Compilers:

• OCaml (using ocamlopt),
• C# (using mcs), and
• Java (using javac).

A subset of these can be installed on Ubuntu (tested on 20.04) via the script ./install-compilers-on-ubuntu-20.04.sh in this repo.

Install Python 3 packages

./install-python-packages.sh

How it works

A benchmark is typically performed as

./benchmark \
    --function-count=$FUNCTION_COUNT \
    --function-depth=$FUNCTION_DEPTH \
    --run-count=5

for suitable values of $FUNCTION_COUNT and FUNCTION_DEPTH or simply

./benchmark

for defaulted values of all the parameters.

A subset of languages combined with set of compilers to benchmark can be chosen as, for instance,

./benchmark --languages=C:tcc,C:gcc-12,C++,D:dmd,D:ldmd2,D:gdc-12,Rust

This will generate code into the directory generated and then, for each combination of language, operation type and compiler, run the supported benchmarks. At the end a Markdown-formatted table showing the results of the benchmark is printed to standard output. Note that the compilation times in this table are titled Time [us/#fn] meaning in unit microseconds normalized with number of test functions generated, that is divided by args.function_count * args.function_depth).

GCC and Clang doesn't perform all semantic checks for C++ (because it's too costly). This is in contrast to D's and Rust's compilers that perform all of them.

Sample generated code

To understand how the code generation works we can, for instance, do

./benchmark --function-count=3 --function-depth=2 --run-count=5

This will, for the C language case, generate a file generated/c/main.c containing

long add_long_n0_h0(long x) { return x + 15440; }
long add_long_n0(long x) { return x + add_long_n0_h0(x) + 95485; }

long add_long_n1_h0(long x) { return x + 37523; }
long add_long_n1(long x) { return x + add_long_n1_h0(x) + 92492; }

long add_long_n2_h0(long x) { return x + 39239; }
long add_long_n2(long x) { return x + add_long_n2_h0(x) + 12248; }


int main(__attribute__((unused)) int argc, __attribute__((unused)) char* argv[]) {
    long long_sum = 0;
    long_sum += add_long_n0(0);
    long_sum += add_long_n1(1);
    long_sum += add_long_n2(2);
    return long_sum;
}

Compiler Object Caches

The numerical constants are randomized using a new seed upon every call. This makes it impossible for any compiler to utilize any caching mechanism upon successive calls with same flags that affect the source generation. The purpose of this is to make the comparison between compilers with no or different different levels of caching more fair.

The caching of the Go reference compiler go, for instance, is effectively disabled by this randomization.

Generics

For each languages $LANG that supports generics an additional templated source file main_t.$LANG will be generated alongside main.$LANG equivalent to the contents of main.$LANG apart from that all functions (except main) are templated. This templated source will be benchmarked aswell. The column Templated in the table below indicates whether or not the compilation is using templated functions.

Conclusions (from sample run shown below)

TCC build speed is varstly superior because of its single-pass code-generation architecture. Partly because parsing the C programming language that doesn’t have to deal with forward declarations and thereby limiting the parsing (and memory allocation) scope to a single function.

The compilers vox dmd are, by a large margin, the fastest. 2 times faster than its closers competitor, tcc. Note that Vox is an experimental language.

The performance of both GCC and Clang gets significanly worse with each new release (currently 8, 9, 10 in the table below).

The templated (generic) C++ source checks about 3 times slower than the non-generic one using gcc-8 but only about 2.3 times slower for gcc-10. For clang++-10 the slowdown is only about 1.6. The corresponding slowdown for generic D (dmd) is about 2.5 times. On the other hand, the generic Rust version interestingly is processed 2-3 times faster than the non-generic version.

Julia's JIT-compiler is (currently) very memory hungry. A maximum recommended product of function-count and function-depth for Julia is 5000. Julia will therefore be excluded from the benchmark when this maximum is reached.

OCaml's optimizing native compiler ocamlopt is very slow for large inputs and is therefore disabled when the product of function-count and function-depth exceeds 10000.

Sample run output

The output on my AMD Ryzen Threadripper 3960X 24-Core Processor running Ubuntu 20.04 for the sample call

./benchmark --function-count=200 --function-depth=200 --run-count=1

results in the following table (copied from the output at the end).

TODO

• Sort table primarily by build time and then check time
• Don’t include Build Time and Build RSS columns when build op is not used.
• Don’t include Check Time and Check RSS columns when check op is not used.
• Add language Fortran.
• Add language Pony.

References

• Go compilation times compared to C++, D, Rust, Pascal (cross-posted)
• LanguageCompilationSpeed