playwright-vs-cypress
playwright-vs-cypress copied to clipboard
a PoC for Playwright vs Cypress in a simple React TS app.
Playwright vs Cypress
This is a PoC for Playwright vs Cypress a simple React TS app.
Table of contents
- Setup
- Playwright
- Cypress
- CI Test results
- Comparison
Setup
After cloning this repo install the project dependencies at the project directory:
npm i
Start the app
npm start
Work around the Playwright MacOS permission issue
This workaround will prevent the dialog "Do you want the application “Chromium.app” to accept incoming network connection?"
Alternatively you can turn off the firewall.
Playwright
The tests are under ./playwright.
In a new tab inside your terminal, run the tests with the following command:
npm run playwright
There are two ways to view the UI while executing Playwright tests:
-
To execute tests with debugger, use Playwright Inspector.
PWDEBUG=1 npm run playwright -
Add
{ headless: false }to chromium launch function in the spec file.
Cypress
The tests are under ./cypress/integration.
To start the test runner:
npm run cypress:open
To run headlessly:
npm run cypress:run
CI Test results
Cypress Dashboard - uses 2 parallel machines.
yml file
Comparison
What matters as of the end of 2022:
- Cypress runs in the browser vs PW runs in Node
- running in the browser has huge advantages
- control browser apis directly (ex: fetch)
- control window events (ex: win.addEventListener)
- control application code directly (ex: app actions, spy/stub problematic application methods)
- running in the browser has huge advantages
- Developer experience / feedback: cy-time-travel-debug vs pw-watch-mode
- syntax: declarative, flow right, terse, fp-like VS imperative, left-hand-side-assignments, jest-like
- component testing: Cy has full support (4 UI frameworks), in browser vs PW has experimental support, in node (React)
| Cypress | Playwright | |
|---|---|---|
| multi-domain, multi-tab scenarios | Multi-domain supported, multi-tab on the road map | supported |
| WebKit support | Supported | supported |
| API testing | Great | supported |
| component testing | supported (React, Vue, Angular, Svelte, NextJs) runs in the browser |
experimental (React) runs in node |
| network handling capabilities | supported Network can be fully stubbed, effectively enabling ui-component-integration testing |
supported |
| point & click test recording (for beginners) | supported | supported |
| local failure diagnosis | Debugging workflow that Cypress provides is still unmatched. The time-traveling and inspecting alongside the command log is an unignorable productivity boost |
supported PW is considering watch-mode which would give a more immediate and visual feedback, vs the VScode plugin or the html report after the test run. |
| access & debug at app source-code level | can use Cypress as dev environment instead of the browser Application action |
n/a |
| community | large | growing |
| training & learning resources | Some of the best documentation online Recipies / patterns with code samples for most known challenges Vast amount of training |
documentation is getting better (some Puppeteer StackOverflow info still applies) |
| target audience | Open source and made for profit, monetized by Cypress Dashboard | Open source, owned by Microsoft |
| reporting | (At this point the paid feature Cypress Dashboard begins to show enterprise perks) Cypress Dashboard |
built-in, Jest-like, and some 3rd party |
| parallelization | CI Load Balancing & Parallelization | local to the machine using workers |
| CI failure diagnosis | artifacts (screenshot, video), CLI code snippets of failure past execution data (failures, flake, commit correlation) |
artifacts (screenshot, video) and CLI Trace viewer |
| CI test flake | Flakey test management Test Muting (ignore test flake you will not address) Test Burn-in (run new tests x times to ensure flake-freeness) |
n/a |
| CI efficiency | Smart orchestration (run failed spec first, cancel run on failure) | n/a |
| analytics | run status & duration test suite size slowest tests top failures |
n/a |
| SSO | Jira Okta Github Enterprise |
n/a |
| tech support | included with Cypress Dashboard sub | n/a |
| automatic waiting | both | |
| video capture, mobile emulation | both | |
| supports Chrome dev protocol | both |