Track benchmarks machines performance difference

[Kerollmops]

trafficstars

We want to check the performance differences between our 4 new benchmark machines. To do so I will create 4 different branches, one for each machine and I will run 4 benchmarks for those 4 branches at the exact same moment, expecting them to be run in parallel on the 4 machines. I decided that running the search_songs benchmarks would be sufficient to see a difference between the machines.

Once the benchmarks are done we will need to take time with @MarinPostma and/or @irevoire to diff the results and display the performance difference. And check that the changes are within an acceptable threshold or not.

Here is the list of benchmarks running on the different machines:

• perf-diff-bench-1 is currently running on sd-168680.
• perf-diff-bench-2 is currently running on sd-168679.
• perf-diff-bench-3 is currently running on sd-168678.
• perf-diff-bench-4 is currently running on sd-168676.

[Kerollmops]

Now that all the benchmarks are done, here is the diff:

./benchmarks/scripts/compare.sh search_songs_perf-diff-bench-1_e10c26e7.json search_songs_perf-diff-bench-2_e10c26e7.json search_songs_perf-diff-bench-3_e10c26e7.json search_songs_perf-diff-bench-4_e10c26e7.json
group                                                 search_songs_perf-diff-bench-1_e10c26e7    search_songs_perf-diff-bench-2_e10c26e7    search_songs_perf-diff-bench-3_e10c26e7    search_songs_perf-diff-bench-4_e10c26e7
-----                                                 ---------------------------------------    ---------------------------------------    ---------------------------------------    ---------------------------------------
smol-songs.csv: asc + default/Notstandskomitee        1.00      3.6±0.02ms        ? ?/sec        1.00      3.5±0.04ms        ? ?/sec        1.09      3.8±0.02ms        ? ?/sec        1.00      3.6±0.02ms        ? ?/sec
smol-songs.csv: asc/Notstandskomitee                  1.04      3.2±0.02ms        ? ?/sec        1.01      3.1±0.01ms        ? ?/sec        1.00      3.1±0.03ms        ? ?/sec        1.06      3.3±0.39ms        ? ?/sec
smol-songs.csv: basic filter: <=/Notstandskomitee     1.04   137.3±14.19µs        ? ?/sec        1.11   146.0±13.31µs        ? ?/sec        1.06    140.0±9.50µs        ? ?/sec        1.00    131.8±6.68µs        ? ?/sec
smol-songs.csv: basic filter: <=/john                 1.00     28.3±0.25µs        ? ?/sec        1.06     30.1±0.36µs        ? ?/sec        1.06     30.0±0.62µs        ? ?/sec        1.09     30.9±0.24µs        ? ?/sec
smol-songs.csv: basic filter: TO/Notstandskomitee     1.13   142.7±10.48µs        ? ?/sec        1.00    125.8±1.07µs        ? ?/sec        1.00    126.4±1.12µs        ? ?/sec        1.02    128.0±0.96µs        ? ?/sec
smol-songs.csv: basic filter: TO/thelonious monk      1.00    136.7±3.24µs        ? ?/sec        1.05    144.0±1.66µs        ? ?/sec        1.01    137.8±3.37µs        ? ?/sec        1.02    139.7±1.55µs        ? ?/sec
smol-songs.csv: desc + default/Notstandskomitee       1.00      3.5±0.05ms        ? ?/sec        1.01      3.5±0.03ms        ? ?/sec        1.11      3.9±0.02ms        ? ?/sec        1.01      3.5±0.03ms        ? ?/sec
smol-songs.csv: desc + default/charles                1.00  1739.1±13.61µs        ? ?/sec        1.05   1830.2±9.91µs        ? ?/sec        1.01   1753.1±9.84µs        ? ?/sec        1.06  1847.2±432.98µs        ? ?/sec
smol-songs.csv: desc/thelonious monk                  1.04      3.6±0.01ms        ? ?/sec        1.00      3.5±0.01ms        ? ?/sec        1.01      3.5±0.01ms        ? ?/sec        1.06      3.7±0.30ms        ? ?/sec
smol-songs.csv: typo/indochien                        1.03    171.2±3.17µs        ? ?/sec        1.06    176.3±4.35µs        ? ?/sec        1.03    170.4±6.93µs        ? ?/sec        1.00    165.7±2.16µs        ? ?/sec
smol-songs.csv: typo/mongus                           1.00    155.1±1.14µs        ? ?/sec        1.01    156.3±1.66µs        ? ?/sec        1.02    158.1±5.39µs        ? ?/sec        1.05    163.0±1.20µs        ? ?/sec

It doesn't seem to be obvious that one machine is faster than another, important diffs are in µs. I need the confirmation of @irevoire and @MarinPostma here.