maplibre-gl-js
maplibre-gl-js copied to clipboard
maplibre
First column in benchmark is faster than the rest
When I ran the benchmark today, it seemed like our current main is performing worse than version 2.3, which was release yesterday. Only two tests affected are Layout and WorkerTrasnfer, and only change since 2.3 are made by dependabot.
Left column is 2.3 ( 91567e98b1abdcdd39c20c77de3e20f2c7f6f80d ), middle column is main right now at fbaebce8e4977c4e06d35e6c54f254ea853d7b1a
In case we get dedicated machines to run CI on then it would be great to have the benchmarks there @hy9be. They don't run in CI at the moment because GitHub runners are too random I think...
Okay, we definitely have to fix this, because now I even doubt if the columns represents the commit hash above. Dedicated machines would be a great step. @wipfli I also noticed you mentioned the xvfb in another context related to unstable github runners, and I thought that maybe we could swap that with X11+dummy or wayland in hopes it would be more stable window server,
I've checked out main and are running benchmarks against it so we can have a better idea about whats going on.
So when I'm running against main it fortunately shows the same, so it's consistent so far.
This is where I checked out 2.3 and ran the benchmarks, so the first and third column should be the same, and the one in the middle is main, which is a bit ahead.
No matter what versions I pick it's always the same - the first is faster in Layout and WorkerTransfer than the rest.
While I think the original issue I opened is still relevant, the issue at hand seems to be related to the browser getting warmed-up and might be solved simply by running the first version twice and reporting only the second run...