Performance has high variance

[kbara]

Performance can vary by around a factor of two on externally-identical runs of the same tools with the same options. What are the detailed reasons, and how can this be mitigated?

[kbara]

GC plays a role in this, especially for the smaller benchmarks.

Wingolog-2 master, with normal GC:

Wingolog-2 master, without GC (forcing a garbage collection, then stopping garbage collection entirely, right outside of the timing loop done in each run):

On the other hand, the largest benchmarks is basically unaffected. Performance increases a bit, but variance does not change much. 1gb with GC:
1gb without GC:

[wingo]

@kbara I think you fixed this when you identified the phi allocation in our bench harness, is that right? I think we can probably close this as invalid. There's still variance but this isn't really on our to-do list right now, seems to me.

[kbara]

We can mark it as low-priority and work on other things, but I think those variance pictures are from after fixing the unsunk CNEWI issue. Here's wingolog-2 from today, showing that it's still an issue:

[wingo]

Those images are from the old filter loop of pflua-bench, which still had a phi allocation. Since merging https://github.com/Igalia/pflua-bench/pull/5 it seems to me that there's no actionable task here. WDYT @kbara? New images:

https://github.com/Igalia/pflua-bench/blob/master/results/1gb-1kb-tcp-port-5555/1gb-1kb-tcp-port-5555.png https://github.com/Igalia/pflua-bench/blob/master/results/ping-flood/ping-flood.png https://github.com/Igalia/pflua-bench/blob/master/results/wingolog.org-1/wingolog.org-1.png https://github.com/Igalia/pflua-bench/blob/master/results/wingolog.org-2/wingolog.org-2.png