cypress
cypress copied to clipboard
Published
20 hours ago •
cypress-io
cypress-io
Failing tests marked as skipped upgrading from 13.1.0 to 13.2.0 (up to 14.1.0) [arm/aarch64]
Update:
I've added a minimal test setup below and it seems like this might be a combination of two things. Starting with 13.2.0, I get an error
1) My First Test
Does not do much and fails!:
Error: EFAULT: bad address in system call argument, write
and when also adding defaultCommandTimeout: 8000 to the config, failing tests are marked as skipped silently.
My First Test
0 passing (8s)
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 2 │
│ Passing: 0 │
│ Failing: 0 │
│ Pending: 0 │
│ Skipped: 2 │
│ Screenshots: 0 │
│ Video: false │
│ Duration: 7 seconds │
│ Spec Ran: spec.cy.js
In 13.1.0, I get the expected response
My First Test
1) Does not do much and fails!
✓ Does not do much! (621ms)
1 passing (14s)
1 failing
1) My First Test
Does not do much and fails!:
AssertionError: Timed out retrying after 8000ms: Expected to find element: `#DoesNotExist`, but never found it.
at Context.eval (webpack:///./cypress/e2e/spec.cy.js:4:7)
Current behavior
After upgrading to version 14.1.0 we noticed failing tests silently being marked as skipped. Once a failing test is executed, that test and all subsequent tests are marked as skipped. Thus CI will always give a green checkmark.
Not making any modifications and running on macOS works as expected, leading me to believe that the problem might have to do with arm/aarch64 but not only that. Now, we are executing the cypress/included docker containers on an embedded device with a custom Yocto distribution built on Ubuntu 18.04.5 LTS. However, as the tests are being run inside the container, I think that has little bearing on the problem. I binary searched through all versions and found that 13.1.0 is the last version that works as expected and 13.2.0 skips tests.
Our tests are very basic, like below. The configuration is without support files or hooks. We start the web server externally so cypress is just pointing to http://localhost:80 via baseUrl.
beforeEach(() => {
cy.visit("/start");
});
it("can enter settings from main", () => {
cy.get("#ExpandButton").click();
// Avoid double clicking by waiting.
cy.wait(500).get("#SettingsButton").click();
cy.get("#OptionsNav");
});
Example with a failing test at the top of the test file:
docker run -it --rm --network host -v /media/data/testing:/e2e -w /e2e --entrypoint bash cypress/included:13.2.0 -c "cypress run"
DevTools listening on ws://127.0.0.1:37863/devtools/browser/6bf62af7-1e97-4868-9df3-6514f5be3bb2
====================================================================================================
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 13.2.0 │
│ Browser: Electron 114 (headless) │
│ Node Version: v20.6.1 (/usr/local/bin/node) │
│ Specs: 1 found (settings.cy.ts) │
│ Searched: cypress**/*.cy.{ts,tsx} │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: settings.cy.ts (1 of 1)
Settings Page
0 passing (12s)
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 3 │
│ Passing: 0 │
│ Failing: 0 │
│ Pending: 0 │
│ Skipped: 3 │
│ Screenshots: 0 │
│ Video: false │
│ Duration: 12 seconds │
│ Spec Ran: settings.cy.ts │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
====================================================================================================
(Run Finished)
Spec Tests Passing Failing Pending Skipped
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ✔ settings.cy.ts 00:12 3 - - - 3 │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
✔ All specs passed! 00:12 3 - - - 3
Example with the same test moved last
docker run -it --rm --network host -v /media/data/testing:/e2e -w /e2e --entrypoint bash cypress/included:13.2.0 -c "cypress run"
DevTools listening on ws://127.0.0.1:46709/devtools/browser/67599640-b306-477d-97b2-8321d5de2ba8
====================================================================================================
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 13.2.0 │
│ Browser: Electron 114 (headless) │
│ Node Version: v20.6.1 (/usr/local/bin/node) │
│ Specs: 1 found (settings.cy.ts) │
│ Searched: cypress**/*.cy.{ts,tsx} │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: settings.cy.ts (1 of 1)
Settings Page
✓ can enter settings from start (3329ms)
✓ can enter settings from main (5257ms)
2 passing (10s)
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 3 │
│ Passing: 2 │
│ Failing: 0 │
│ Pending: 0 │
│ Skipped: 1 │
│ Screenshots: 0 │
│ Video: false │
│ Duration: 10 seconds │
│ Spec Ran: settings.cy.ts │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
====================================================================================================
(Run Finished)
Spec Tests Passing Failing Pending Skipped
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ✔ settings.cy.ts 00:10 3 2 - - 1 │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
✔ All specs passed! 00:10 3 2 - - 1
Using exactly the same test files with cypress/included:13.1.0, the results are as expected
docker run -it --rm --network host -v /media/data/testing:/e2e -w /e2e --entrypoint bash cypress/included:13.1.0 -c "cypress run"
DevTools listening on ws://127.0.0.1:38371/devtools/browser/46e0f31f-5c0d-4a13-8a19-ffdf1fba74dd
[434:0228/082302.734027:ERROR:gpu_memory_buffer_support_x11.cc(44)] dri3 extension not supported.
====================================================================================================
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 13.1.0 │
│ Browser: Electron 106 (headless) │
│ Node Version: v20.5.0 (/usr/local/bin/node) │
│ Specs: 1 found (settings.cy.ts) │
│ Searched: cypress**/*.cy.{ts,tsx} │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: settings.cy.ts (1 of 1)
Settings Page
✓ can enter settings from start (3681ms)
✓ can enter settings from main (5273ms)
(Attempt 1 of 3) can render setup settings page
(Attempt 2 of 3) can render setup settings page
1) can render setup settings page
2 passing (1m)
1 failing
1) Settings Page
can render setup settings page:
AssertionError: Timed out retrying after 8000ms: Expected to find element: `#WifiHotspotParameters2`, but never found it.
at Context.eval (webpack:///./cypress/e2e/settings.cy.ts:30:0)
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 3 │
│ Passing: 2 │
│ Failing: 1 │
│ Pending: 0 │
│ Skipped: 0 │
│ Screenshots: 3 │
│ Video: false │
│ Duration: 1 minute, 6 seconds │
│ Spec Ran: settings.cy.ts │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
(Screenshots)
- /e2e/cypress/screenshots/settings.cy.ts/Settings Page -- can render setup settin (1280x720)
gs page (failed).png
- /e2e/cypress/screenshots/settings.cy.ts/Settings Page -- can render setup settin (1280x720)
gs page (failed) (attempt 2).png
- /e2e/cypress/screenshots/settings.cy.ts/Settings Page -- can render setup settin (1280x720)
gs page (failed) (attempt 3).png
====================================================================================================
(Run Finished)
Spec Tests Passing Failing Pending Skipped
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ✖ settings.cy.ts 01:06 3 2 1 - - │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
✖ 1 of 1 failed (100%) 01:06 3 2 1 - -
Desired behavior
No response
Test code to reproduce
cypress.config.js:
const { defineConfig } = require('cypress')
module.exports = defineConfig({
projectId: 'Test',
e2e: {
baseUrl: 'http://localhost:80',
specPattern: "cypress/e2e/**/*.cy.{js,jsx}",
supportFile: false,
},
defaultCommandTimeout: 8000,
})
cypress/e2e/spec.cy.js:
describe('My First Test', () => {
it('Does not do much and fails!', () => {
cy.visit("/")
cy.get("#DoesNotExist")
})
it('Does not do much!', () => {
cy.visit("/")
})
})
Cypress Version
14.1.0
Node version
The versions that come with the cypress/included docker containers
Operating System
Ubuntu 18.04.5 LTS on aarch64
Debug Logs
I can add these but not sure if I should add the results from 13.2.0 or 14.1.0.
Other
No response
