[Feature]: HTTP status code constants

[LinkPovilas]

Current behavior

If an attempt is made to update playwright-webkit to the latest version in Cypress source code, it fails the Cypress test spec

• packages/driver/cypress/e2e/commands/cookies.cy.js

which is part of driver-integration-tests-webkit.

This is reproducible on Ubuntu with versions of playwright-webkit starting with 1.37.0 and extending to the current version (with WebKit ~~17.4~~ 18.0).

Cypress is currently configured to use [email protected] released in July 2022:

https://github.com/cypress-io/cypress/blob/d8884e0a342e92a41fb3e187a807ae7babab3cc7/package.json#L189

Desired behavior

WebKit should pass all tests from driver-integration-tests-webkit when using the latest version of playwright-webkit.

• Qualify if this is due to bug(s) in Playwright/WebKit or Cypress
• Depending on outcome, report bugs to Playwright and / or fix bugs in Cypress

Test code to reproduce

npm install yarn@latest -g
git clone https://github.com/cypress-io/cypress
cd cypress
yarn add playwright-webkit@latest -D -W
yarn playwright install-deps webkit # Linux only

then

yarn workspace @packages/driver cypress:run --browser webkit --spec cypress/e2e/commands/cookies.cy.js

or

yarn workspace @packages/driver cypress:open --e2e --browser webkit

and select cypress/e2e/commands/cookies.cy.js to run in UI.

Playwright Version

First reported on [email protected] Continues to be reproducible on [email protected]

Cypress Version

First reported on 13.13.2 Continues to be reproducible on 14.1.0

Node version

~~v18.17.1~~ v20.18.1

Operating System

~~Ubuntu 22.04.4 LTS~~ Ubuntu 22.04.5 LTS and Ubuntu 24.04.2 LTS

Debug Logs

See failures in CircleCI at https://app.circleci.com/pipelines/github/cypress-io/cypress/63152/workflows/e49a5390-7dc4-432e-abce-391b205c080b/jobs/2615104

  1) src/cy/commands/cookies - no stub
       #getCookies
         returns cookies for the specified domain:

      AssertionError: expected 'key2' to equal 'key1'
      + expected - actual

      -'key2'
      +'key1'

      at  ([native code])
      at  (http://www.barbaz.com:3500/__cypress/tests:1717:41)
  getRet@
  tryCatcher@
  @
  thenFn@
  @
  tryCatcher@
  @
  @
  @
  _drainQueueStep@
  _drainQueue@
  @
  @

  2) src/cy/commands/cookies - no stub
       #getCookie
         returns the cookie from the specified domain:

      AssertionError: expected 'foobar.com value' to equal 'www.foobar.com value'
      + expected - actual

      -'foobar.com value'
      +'www.foobar.com value'

      at  ([native code])
      at  (http://www.barbaz.com:3500/__cypress/tests:1870:38)
  getRet@
  tryCatcher@
  @
  thenFn@
  @
  tryCatcher@
  @
  @
  @
  _drainQueueStep@
  _drainQueue@
  @
  @

  3) src/cy/commands/cookies
       #setCookie
         can set cookies with sameSite:

      AssertionError: expected { Object (name, value, ...) } to have property 'sameSite' of 'no_restriction', but got 'lax'
      + expected - actual

      -'lax'
      +'no_restriction'

      at  ([native code])
  applyChainer@
  arrayReduce@
  reduce@
  applyChainers@
  tryCatcher@
  @
  shouldFn@
  @
  tryCatcher@
  @
  @
  @
  _drainQueueStep@
  _drainQueue@
  @
  @

Other

• See WebKit (Experimental)
• https://playwright.dev/docs/browsers#webkit
• See https://github.com/cypress-io/cypress/pull/29955 for failed attempt to update.

[MikeMcC399]

Further test results which may assist troubleshooting:

Results of yarn workspace @packages/driver cypress:run --browser webkit --spec cypress/e2e/commands/cookies.cy.js dependent on version of playwright-webkit

Playwright	pass	fail	WebKit	Notes
1.24.2	129	0	16.0	(1)
1.34.3	129	0	16.4	(2)
1.35.0	129	0	16.4	(3)
1.36.2	129	0	17.0	(3)
1.37.0	128	1	17.0	(4)
1.37.1	127	2	17.0	(5)
1.38.1	127	2	17.0	(5)
1.39.0	127	2	17.4	(5)
1.40.1	128	1	17.4	(4)
1.41.2	127	2	17.4	(5)
1.42.1	127	2	17.4	(5)
1.43.1	127	2	17.4	(5)
1.44.1	127	2	17.4	(5)
1.45.0	127	2	17.4	(6)
1.45.1	126	3	17.4	(7)
1.45.3	126	3	17.4	(7)
1.46.0	127	3	18.0	(7)

(1) Currently configured version (2) Previously recommended version (3) Test does not exit and instead wrongly opens Cypress runner at end in global mode (4) Failure is AssertionError: expected 'foobar.com value' to equal 'www.foobar.com value' (5) Failures as (4) above, additionally AssertionError: expected 'key2' to equal 'key1' (6) Failure as (4) above, additionally AssertionError: expected { Object (name, value, ...) } to have property 'sameSite' of 'no_restriction', but got 'lax' (7) Failures as (4), (5) & (6) above

Change version with for instance:

yarn add [email protected] -D -W
yarn playwright install-deps webkit

Test with:

yarn workspace @packages/driver cypress:run --browser webkit --spec cypress/e2e/commands/cookies.cy.js

View WebKit version with

yarn workspace @packages/driver cypress:open --e2e --browser webkit

Settings > Project settings > Resolved configuration (Fails to launch Cypress Runner with some old versions)

For debug logs use:

export DEBUG=cypress:server:browsers:webkit-automation

[MikeMcC399]

Since there hasn't been any progress made on this issue and a minimum version of [email protected] is required to support Ubuntu 24.04, I suggest to modify packages/driver/cypress/e2e/commands/cookies.cy.js to skip tests that fail in current versions of WebKit.

There are already multiple exceptions made in this test code depending on whether WebKit is being tested.

[MikeMcC399]

@jennifer-shehane

3 out of 129 cookie tests fail for WebKit installed from [email protected] and above

I would submit a PR that skips these tests for WebKit if you agree. This would unblock an upgrade to the latest version of playwright-webkit and the system tests would be able to run under Debian 12 and Ubuntu 24.04.

Let me know if you want a PR for this.

Edit: I may have been too optimistic as it appears that there are other tests that fail apart from the cookies test when an upgrade is attempted. I'll take a look at doing smaller steps and see if anything looks feasible.

[MikeMcC399]

WebKit 26.0 is now the current version. If an upgrade is attempted, tests are still failing as before.

[AtofStryker]

@MikeMcC399 I'm looking into updating WebKit to v26. I've downloaded playwright-webkit 1.56.1 and ran yarn workspace @packages/driver cypress:run --browser webkit --spec cypress/e2e/commands/cookies.cy.js. Oddly enough, the spec passes? I am wondering if you are seeing the same

[AtofStryker]

ah there are inconsistencies in the linux and mac versions of webkit, in particular the ordering of the cookies in the context. My PR will account for this

[MikeMcC399]

@AtofStryker

@MikeMcC399 I'm looking into updating WebKit to v26. I've downloaded playwright-webkit 1.56.1 and ran yarn workspace @packages/driver cypress:run --browser webkit --spec cypress/e2e/commands/cookies.cy.js. Oddly enough, the spec passes? I am wondering if you are seeing the same

The steps to reproduce continue to fail on Ubuntu 24.04.3 with 1.56.1.

I see though that you've progressed further and are taking account of it in PR https://github.com/cypress-io/cypress/pull/32852

[cypress-bot[bot]]

Released in 15.6.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to Cypress v15.6.0, please open a new issue.