mapbox-gl-js icon indicating copy to clipboard operation
mapbox-gl-js copied to clipboard

Running CircleCI render tests on Windows, Mac and Firefox

Open SnailBones opened this issue 1 year ago • 3 comments

Closes https://github.com/mapbox/gl-js-team/issues/480

Adds three new runs in CircleCI:

  • test-render-macos-chrome
  • test-render-windows-chrome
  • test-render-linux-firefox

Windows and Mac tests are running on virtual machines.

Failing tests on Windows and Mac are due to bugs including https://github.com/mapbox/mapbox-gl-js/issues/7331. Using --use-gl-desktop lets these tests pass on Mac but also causes missing globes, inconsistent stars and fill extrusion patterns (as was the reason for disabling this locally in https://github.com/mapbox/mapbox-gl-js/pull/12090).

The same tests skipped on Windows and Mac formerly formerly also fail on Linux without the --use-gl-desktop flag (added in https://github.com/mapbox/mapbox-gl-js/pull/10389) as seen in this run without the flag.

I've taken the following approach to updating "allowed" properties:

  • If the images are visually indistinguishable and dif is suitably low (< 1% for small images and less for larger or detail-oriented ones), raise the "allowed" threshold just above the dif.
  • Otherwise, add to a platform-specific ignore file.
  • For cases where appearance is still good but the dif is higher, we may want to consider introducing platform-specific "allowed" values, but I've avoided that for now to minimize the scope of this PR.

This currently makes CI runs take longer. Previous runs tended to finish in 5-6 minutes. All of the new runs added take longer:

  • Firefox: ~8m
  • Mac: ~14m
  • Windows: ~20m

There's likely room to optimize the CI config through caching. but 90% of the time spent on Windows is running tests. A more effective way to speed up CI outputs would be splitting it into two (or more) runs, with each running half the tests.

Launch Checklist

  • [x] Operating systems
    • [x] Windows
    • [x] MacOS
  • [x] Browsers:
    • [x] Firefox
  • [x] Update (default?) "allowed" values so tests pass
  • [ ] Open issues tracking all platform-specific test failures
  • [ ] Cache installed browsers to improve test performance

SnailBones avatar Sep 29 '22 23:09 SnailBones