quick-lint
Run precise benchmarks on CI
We could run some precise benchmarks on CI. Examples:
- Maximum memory usage (ASLR should be disabled)
- Instruction count (ASLR should be disabled?)
- Cycle count (ASLR should be disabled) (needs a CPU emulator)
Valgrind might help a lot.
Also, we need to store the collected data somewhere, and have some fancy dashboards to present the data.
Memory usage:
Valgrind's memcheck tool provides some details, but not a high-water mark:
==14522== HEAP SUMMARY:
==14522== in use at exit: 130,789 bytes in 182 blocks
==14522== total heap usage: 791 allocs, 609 frees, 211,956 bytes allocated
Instruction count:
Valgrind can provide this information. I think we can run the Cachegrind tool and track its global statistics. We need to specify cache sizes and such; I think the default is inherited from the host, making the results non-portable. (But libc versions also play a factor in portability...)
Database:
We could use a Git repository as a database. GitHub Actions should be able to create a Git commit containing JSON files with the measurements. We could make a program which extracts data from this Git repository and generates charts. Hacky, but easy to implement and debug.
Database:
I set up an SFTP/HTTPS server for builds: https://c.quick-lint-js.com/builds/ We could use this server to store benchmark results too. Probably easier to deal with than Git.
Deno has some precise benchmarks which could inspire us: https://deno.land/benchmarks