playcanvas
Proposal for E2E (visual) regression test suite
Motivation
It is possible that additional features, specification changes, etc. will cause the rendering results to be different from what they were before. For this reason, we propose the introduction of an E2E (visual) test environment. This enables visual regression testing by comparing a snapshot of the rendered image with the correct image using headless rendering by Chrome. This mechanism will allow early detection of regression bugs that cannot be covered by unit test. Passing this E2E (Visual) Test guarantees that compatibility is maintained, allowing for major engine refactoring without concern. Since it is headless execution, this mechanism can be incorporated into CI, such as Github Actions.
Existing tools
The Jest test library has an extension called jest-image-snapshot, which compares the results of the correct image and current drawings and treats any differences as errors. It also outputs the difference images. It is also possible to set a threshold for how much difference is acceptable.
There is a port of jest-image-snapshot for non-Jest test libraries (vitest and mocha).
(I have used both the original jest-image-snapshot and the port for vitest with success on my RhodoniteTS. see the following screen capture).
Since PlayCanvas already seems to use mocha for unit tests, you might want to use the porting for mocha. https://sebastianlandwehr.com/blog/image-snapshot-testing-with-mocha/
Things to consider
- It would make sense to use existing examples that already exist as visual test items.
- It may be necessary to add a small code to the existing examples that notifies mocha that the rendering is complete.
- However, no other changes will probably be necessary, and the existing examples structure can be appropriated as is.
- Whether to run visual tests with the GPU enabled or in software emulation depends on your objectives.
- Software emulation can currently test only the WebGL version, but the results will be the same in all environments, so if you want to test locally as well as on CI, software emulation is the way to go.
- On the other hand, GPU-enabled visual tests can also test WebGPU, but the results will vary slightly depending on the execution environment, so they are basically for CI execution only, and require either a GPU-enabled instance of Github or a dedicated machine and a Self Hosted Github Action.
Thought?
This is definitely great!
Sometimes the change is intentional, so it would be great to ensure that visual tests don't fail, but provide an insight - to what have changed, as some changes might be expected.
To this end, I think it is important to limit the changes in PR to a single function. Still, there may be times when human judgment is needed to determine whether a test failure is due to an intentional specification change or an unintentional bug. Anyway, in this mechanism, you can save the current rendering as the correct image by running the 'update' command.
I created PR. Please check it out https://github.com/playcanvas/engine/pull/7705