playwright-vs-cypress icon indicating copy to clipboard operation
playwright-vs-cypress copied to clipboard

a PoC for Playwright vs Cypress in a simple React TS app.

Playwright vs Cypress

This is a PoC for Playwright vs Cypress a simple React TS app.

Table of contents

  • Setup
  • Playwright
  • Cypress
  • CI Test results
  • Comparison


Setup

After cloning this repo install the project dependencies at the project directory:

npm i

Start the app

npm start
Work around the Playwright MacOS permission issue


This workaround will prevent the dialog "Do you want the application “Chromium.app” to accept incoming network connection?"

Alternatively you can turn off the firewall.


Playwright

The tests are under ./playwright.

In a new tab inside your terminal, run the tests with the following command:

npm run playwright

There are two ways to view the UI while executing Playwright tests:

  1. To execute tests with debugger, use Playwright Inspector.

    PWDEBUG=1 npm run playwright
    
  2. Add { headless: false } to chromium launch function in the spec file.


Cypress

The tests are under ./cypress/integration.

To start the test runner:

npm run cypress:open

To run headlessly:

npm run cypress:run


CI Test results

Cypress Dashboard - uses 2 parallel machines.

Github Actions

yml file


Comparison

What matters as of the end of 2022:

  • Cypress runs in the browser vs PW runs in Node
    • running in the browser has huge advantages
      • control browser apis directly (ex: fetch)
      • control window events (ex: win.addEventListener)
      • control application code directly (ex: app actions, spy/stub problematic application methods)
  • Developer experience / feedback: cy-time-travel-debug vs pw-watch-mode
  • syntax: declarative, flow right, terse, fp-like VS imperative, left-hand-side-assignments, jest-like
  • component testing: Cy has full support (4 UI frameworks), in browser vs PW has experimental support, in node (React)
Cypress Playwright
multi-domain, multi-tab scenarios Multi-domain supported, multi-tab on the road map supported
WebKit support Supported supported
API testing Great supported
component testing supported (React, Vue, Angular, Svelte, NextJs)
runs in the browser
experimental (React)
runs in node
network handling capabilities supported
Network can be fully stubbed, effectively enabling ui-component-integration testing
supported
point & click test recording (for beginners) supported supported
local failure diagnosis Debugging workflow that Cypress provides is still unmatched.
The time-traveling and inspecting alongside the command log is an unignorable productivity boost
supported PW is considering watch-mode which would give a more immediate and visual feedback, vs the VScode plugin or the html report after the test run.
access & debug at app source-code level can use Cypress as dev environment instead of the browser
Application action
n/a
community large growing
training & learning resources Some of the best documentation online
Recipies / patterns with code samples for most known challenges
Vast amount of training
documentation is getting better (some Puppeteer StackOverflow info still applies)
target audience Open source and made for profit, monetized by Cypress Dashboard Open source, owned by Microsoft
reporting (At this point the paid feature Cypress Dashboard begins to show enterprise perks)
Cypress Dashboard
built-in, Jest-like, and some 3rd party
parallelization CI Load Balancing & Parallelization local to the machine using workers
CI failure diagnosis artifacts (screenshot, video), CLI
code snippets of failure
past execution data (failures, flake, commit correlation)
artifacts (screenshot, video) and CLI
Trace viewer
CI test flake Flakey test management

Test Muting (ignore test flake you will not address)
Test Burn-in (run new tests x times to ensure flake-freeness)
n/a
CI efficiency Smart orchestration (run failed spec first, cancel run on failure) n/a
analytics run status & duration
test suite size
slowest tests
top failures
n/a
SSO Jira
Okta
Github Enterprise
n/a
tech support included with Cypress Dashboard sub n/a
automatic waiting both
video capture, mobile emulation both
supports Chrome dev protocol both