Benchmark across different branches

[technicalpickles]

trafficstars

When I'm working on performance, I tend to want to compare a baseline from our main branch against my current work. It'd be pretty handy to be able to specify branches at the CLI level to handle the switch. Maybe:

hyperfine --compare-branch main "rails runner true"

You can achieve something similar by including the git checkouts in the command:

$ hyperfine "git switch main; DISABLE_SPRING=1 bin/rails runner true" "git switch performance-improvements; DISABLE_SPRING=1 bin/rails runner true"
Benchmark 1: git checkout main; DISABLE_SPRING=1 bin/rails runner true
  Time (mean ± σ):      8.690 s ±  0.876 s    [User: 4.835 s, System: 4.665 s]
  Range (min … max):    8.114 s … 11.106 s    10 runs

Benchmark 2: git switch performance-improvements; DISABLE_SPRING=1 bin/rails runner true
  Time (mean ± σ):      8.352 s ±  0.363 s    [User: 4.778 s, System: 4.621 s]
  Range (min … max):    7.950 s …  9.020 s    10 runs

Summary
  'git switch performance-improvements; DISABLE_SPRING=1 bin/rails runner true' ran
    1.04 ± 0.11 times faster than 'git checkout main; DISABLE_SPRING=1 bin/rails runner true'

[sharkdp]

You can do this using a combination of --parameter-list and --setup. Something like the following should work:

hyperfine \
  --parameter-list branch main,performance-improvements \
  --setup "git switch {branch}" \
  "rails runner true"

When I'm working on performance, I tend to want to compare a baseline from our main branch against my current work.

This is indeed a very common use case. We should probably write some documentation for that.

[nirs]

Another way to do this is to use git worktree to have both main/master branch and your work branch checked out and built and run the benchmark with both. This way you can run benchmarks with uncommitted changes.