Cypress test optimizations

[jrtashjian]

This PR is an attempt to optimize our Cypress test suites by focusing the tests around interacting within the editor.

These comparisons are based on running both the Features and Feature (child block) blocks end-to-end tests together.

Currently for each block's test we are taking these steps:

1. Clear editor.
2. Insert block being tested.
3. Perform interactions and assertions.
4. Save page.
5. Check for block errors.
6. View page on front-end (page load).
7. Perform assertions.
8. Go back to the edit screen (page load).

Average time to complete: 31 seconds

Here are the steps this PR takes:

1. Clear editor.
2. Insert block being tested.
3. Perform interactions and assertions.
4. Check for block errors.

Average time to complete: 17 seconds

I've removed the front-end check because we were only checking that classes or styles existed in the HTML. This is redundant because our unit tests using Jest already test that a block's HTML output is what we expect given specific attribute values. Instead of testing the HTML output, what we really want is a visual test to make sure things are visually consistent. Because of this I've added the cypress-plugin-snapshots plugin to add visual regression detection.

Visual Regression Tests

At the end of each test suite I've added a test that makes a snapshot of the block viewed in the theme. This initial snapshot would be the baseline. Every subsequent run of the test suite will create a new snapshot which will be compared with this baseline. If any changes are detected the test will throw an error and provide both the new snapshot image and a diff image off the original.

Average time to complete: 23 seconds (with optimizations above)

Additional Improvements

These are some things we should look into for improving our tests and making CoBlocks more robust in different environments.

• [ ] When inserting the block being tested, don't clear the editor. We should have a snapshot where each block interaction we're testing. We need to find a way to insert a block in our beforeEach call and allow the test to reference it without any confusing selectors. We should be able to do this with a variable we just overwrite with the new DOM element.

• [ ] The ability to test against the latest WordPress Core as well as with the latest Gutenberg Plugin. This will allow us to catch visual regressions as new features in Gutenberg are released in the plugin and maintain stability with Core.

• [ ] The ability to test for visual regressions against different themes we want 100% support for: Go, TwentyTwenty, etc.

New Problems

• [ ] We need multiple environments created to run these tests against.

1. The latest WordPress Core.
2. The latest WordPress Core with the Gutenberg plugin active.
3. The latest default theme included with Core.
4. The latest version of our Go theme.
5. The latest of any other themes we need 100% support for.
6. A theme with absolutely zero front-end styles (to ensure our styles are theme agnostic).

• [ ] We need to ensure snapshots are created for each environment. If we only have one snapshot stored with the block and we change the theme, the snapshot test will fail.

• [ ] Increased running time. Although this PR is for optimizing in an effort to make our tests quicker, we are adding more and more tests. This is important but they do take time. This is most likely just a case of increasing parallelism on CircleCI and/or Cypress.

[cypress[bot]]

---

Test summary

10 • 0 • 0 • 0

---

Run details

Project	CoBlocks
Status	Passed
Commit	3de1ec4a10
Started	Jun 2, 2020 9:46 PM
Ended	Jun 2, 2020 9:47 PM
Duration	01:05 💡
OS	Linux Debian - 10.4
Browser	Multiple

View run in Cypress Dashboard ➡️

---

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

[jrtashjian]

Closing this as we may be moving away from Cypress.