sightglass icon indicating copy to clipboard operation
sightglass copied to clipboard

sightglass-next: extract native baseline benchmarking from webui_runner

Open abrown opened this issue 4 years ago • 1 comments

One useful feature of the original sightglass code was the ability to run the benchmarks as native machine code in order to form a baseline for comparison. If we migrate this functionality from webui_runner to benchmarks-next (e.g.), we can then fully replace the old sightglass runner with the new one.

This involves some investigation to determine how to hook into the bench_start() and bench_end() calls with perf.

abrown avatar Mar 05 '21 00:03 abrown

I've looked into this a bit more and it seems highly unlikely that we will be able to trigger perf to measure hardware events (e.g. instructions retired) using tracepoints. I was hoping that something like the following would work:

clang benchmark.c -O3 -g -I. -o benchmark
perf probe -x ./benchmark bench_start=_bench_start
perf probe -x ./benchmark bench_end=_bench_end
perf record -e cycles --filter "start probe_benchmark:bench_start, stop probe_benchmark:bench_end" ./benchmark

Unfortunately, the last call fails because perf does not measure hardware events using these types of triggers; the perf-record documentation points out that the --filter "option should follow an event selector (-e) which selects either tracepoint event(s) or a hardware trace PMU (e.g. Intel PT or CoreSight)".

Another option is to start measuring directly with perf_event_open inside the bench_start call, much like what happens in the recorder crate. We could modify the sightglass-next.h header to do this when compiling the benchmarks natively (i.e. not to Wasm) so that we could get at least some native measurements.

abrown avatar Apr 06 '21 21:04 abrown