compiler-benchmark

compiler-benchmark copied to clipboard

Hi, thanks for the project. Consider to add extra compilers for C: Zig and D.

I've tried to generate C source from your repo (--function-count=200 --function-depth=200).

Then compile it with all available on my system compilers. It seems working, but some issues should be solved (maybe with some additional flags for compiler):

• Zig is very slow at first compilation. Moreover it is caching the results so the next compilation is very fast (some miliseconds).
• It seems Zig makes stripped version by default (which also could take time). Becase it is the only exectubale file that not affected by "strip" command at all - the size is not changing.
• D ImportC compiling your functions, but hardly could be consider as fair-play participant without support of preprocessor. And when I try to add "#include <stdio.h>" - it also gave me an error. So I'm not sure how properly measure lack of this "features".

I've used hyperfine on my Manjaro laptop:

• AMD Ryzen 3 5300U
• gcc (GCC) 12.2.0
• clang version 14.0.6
• zig 0.9.1
• DMD64 D Compiler v2.100.2

Results: Benchmark 1: clang -o code_clang code.c Time (abs ≡): 5.034 s [User: 4.902 s, System: 0.124 s]

Benchmark 2: gcc -o code_gcc code.c Time (abs ≡): 12.375 s [User: 12.024 s, System: 0.326 s]

Benchmark 3: zig cc -o code_zig code.c Time (abs ≡): 70.219 s [User: 68.940 s, System: 1.083 s]

Benchmark 4: dmd -of=code_dmd code.c Time (abs ≡): 1.344 s [User: 1.127 s, System: 0.211 s]

Summary 'dmd -of=code_dmd code.c' ran 3.75 times faster than 'clang -o code_clang code.c' 9.21 times faster than 'gcc -o code_gcc code.c' 52.26 times faster than 'zig cc -o code_zig code.c'

How can dmd be so fast??

How can dmd be so fast??

I think the real question is: How can GCC and Clang be so slow!? And if you think those are slow, try Swift (via swiftc on Linux); it's CPU usage is insane, Jonathan Blow calls such languages "big agenda languages". Especially when you consider the out-of-this-world speed of tinyc both in time and space.

I think one answer is that our CPUs have gotten so fast that nobody cares or see the need to question the performance of our bloated non-integrated (separate linking phase) compiler architectures actually are much slower than they need to be.

If you think dmd is fast, try tinycc:

apt install tcc

followed by

./benchmark --languages=D:dmd,C:tcc --function-count=200 --function-depth=200

giving

# Benchmark:
- Generating generated/d/main_0.d took 0.362 seconds (D)
- Check took 0.268 seconds (using "dmd" version v2.101.0-rc.1-64-g4dc1e98a7f)
- Generating generated/d/main_t_0.d took 0.362 seconds (D)
- Check took 0.760 seconds (using "dmd" version v2.101.0-rc.1-64-g4dc1e98a7f)
- Generating generated/c/main_0.c took 0.312 seconds (C)
- Check took 0.062 seconds (using "/home/per/.local/tinycc-snapshot/bin/tcc" version 0.9.27)
- Compile took 0.675 seconds (using "dmd" version v2.101.0-rc.1-64-g4dc1e98a7f)
- Compile took 1.249 seconds (using "dmd" version v2.101.0-rc.1-64-g4dc1e98a7f)
- Compile took 0.061 seconds (using "/home/per/.local/tinycc-snapshot/bin/tcc" version 0.9.27)
- Build took 0.873 seconds (using "dmd" version v2.101.0-rc.1-64-g4dc1e98a7f)
- Build took 1.454 seconds (using "dmd" version v2.101.0-rc.1-64-g4dc1e98a7f)
- Build took 0.071 seconds (using "/home/per/.local/tinycc-snapshot/bin/tcc" version 0.9.27)

| Lang-uage | Temp-lated | Check Time [us/fn] | Compile Time [us/fn] | Build Time [us/fn] | Run Time [us/fn] | Check RSS [kB/fn] | Build RSS [kB/fn] | Exec Version | Exec Path | 
| :-------: | ---------- | :----------------: | :------------------: | :----------------: | :--------------: | :---------------: | :---------------: | :----------: | :-------: | 
| D         | No         |    6.7 (4.3x)      |   16.9 (11.0x)       |   21.8 (12.3x)     |     49 (3.2x)    |    4.3 (9.2x)     |   13.5 (29.1x)    | v2.101.0-rc.1-64-g4dc1e98a7f | dmd       | 
| D         | Yes        |   19.0 (12.3x)     |   31.2 (20.3x)       |   36.4 (20.5x)     |     48 (3.1x)    |   12.5 (26.9x)    |   22.2 (47.8x)    | v2.101.0-rc.1-64-g4dc1e98a7f | dmd       | 
| C         | No         |    1.5 (best)      |    1.5 (best)        |    1.8 (best)      |     15 (best)    |    0.5 (best)     |    0.5 (best)     | 0.9.27       | tcc       | 

. Please note the 30x reduction memory usage for tcc compared to dmd and that its code execute 3x faster than dmds! In debug mode this is.

"There's plenty of room at the bottom" even for dmd, to quote Feynman...

Note however that the C programming language doesn't support forward declarations and templates of course and other stuff so the compiler doesn't need to do more than on pass and and doesn't even need to keep an AST around so it can basically be implemented in a stack-machine that generates machine code in a stream-like fasion which is likely one of the key reasons why it require so little resources. Kind of like what a (portable) assembler does. If computers hadn't been that slow back the architecture of the language would like had required a compiler requiring more resources

I believe Jai is supposed to be lightning fast aswell but I haven't acquired a beta yet.

Interesting. Thank you for doing these benchmarks! 🇸🇪

How can dmd be so fast??

Ask Walter bright and start attending DLang confs and meetings.

How can dmd be so fast??

Ask Walter bright and start attending DLang confs and meetings.

Immo was fired from his previous company and now he stopped using D

Immo was fired from his previous company and now he stopped using D

I have some faith in Zig and Rust aswell.