mapbox-gl-js
mapbox-gl-js copied to clipboard
Running CircleCI render tests on Windows, Mac and Firefox
Closes https://github.com/mapbox/gl-js-team/issues/480
Adds three new runs in CircleCI:
- test-render-macos-chrome
- test-render-windows-chrome
- test-render-linux-firefox
Windows and Mac tests are running on virtual machines.
Failing tests on Windows and Mac are due to bugs including https://github.com/mapbox/mapbox-gl-js/issues/7331. Using --use-gl-desktop
lets these tests pass on Mac but also causes missing globes, inconsistent stars and fill extrusion patterns (as was the reason for disabling this locally in https://github.com/mapbox/mapbox-gl-js/pull/12090).
The same tests skipped on Windows and Mac formerly formerly also fail on Linux without the --use-gl-desktop
flag (added in https://github.com/mapbox/mapbox-gl-js/pull/10389) as seen in this run without the flag.
I've taken the following approach to updating "allowed" properties:
- If the images are visually indistinguishable and dif is suitably low (< 1% for small images and less for larger or detail-oriented ones), raise the "allowed" threshold just above the dif.
- Otherwise, add to a platform-specific ignore file.
- For cases where appearance is still good but the dif is higher, we may want to consider introducing platform-specific "allowed" values, but I've avoided that for now to minimize the scope of this PR.
This currently makes CI runs take longer. Previous runs tended to finish in 5-6 minutes. All of the new runs added take longer:
- Firefox: ~8m
- Mac: ~14m
- Windows: ~20m
There's likely room to optimize the CI config through caching. but 90% of the time spent on Windows is running tests. A more effective way to speed up CI outputs would be splitting it into two (or more) runs, with each running half the tests.
Launch Checklist
- [x] Operating systems
- [x] Windows
- [x] MacOS
- [x] Browsers:
- [x] Firefox
- [x] Update (default?) "allowed" values so tests pass
- [ ] Open issues tracking all platform-specific test failures
- [ ] Cache installed browsers to improve test performance