start-server-and-test
start-server-and-test copied to clipboard
bahmutov
Running with Cypress is flaky in CircleCi
Goto https://github.com/bahmutov/start-server-and-test/issues/250#issuecomment-1079451745 to see how I resolved this
Running Cypress will intermittently fail in CircleCI. I haven't been able to identify a pattern that causes it, my guess is it's some kind of race condition.
Running yarn cy:ci locally works perfectly every time.
in package.json:
"scripts": {
...
"start": "ng serve",
"cy:run": "$(yarn bin)/cypress run",
"cy:ci": "start-server-and-test start http-get://localhost:4200 cy:run",
...
},
...
"devDependencies": {
...
"cypress": "^4.0.0",
"cypress-browser-extension-plugin": "^0.1.0",
"cypress-iframe": "^1.0.1",
"start-server-and-test": "^1.10.11",
...
},
in .circleci/config.yml:
- run:
name: 'Run Cypress tests'
command: yarn cy:ci
circle logs:
#!/bin/bash -eo pipefail
yarn cy:ci
yarn run v1.22.4
$ start-server-and-test start http-get://localhost:4200 cy:run
starting server using command "npm run start"
and when url "[ 'http-get://localhost:4200' ]" is responding with HTTP status code 200
running tests using command "npm run cy:run"
> [email protected] start /home/circleci/repo
> ng serve
... // App builds
** Angular Live Development Server is listening on localhost:4200, open your browser on http://localhost:4200/ **
: Compiled successfully.
> [email protected] cy:run /home/circleci/repo
> $(yarn bin)/cypress run
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] cy:run: `$(yarn bin)/cypress run`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] cy:run script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /home/circleci/.npm/_logs/2020-04-15T21_27_08_984Z-debug.log
Error: Command failed with exit code 1: npm run cy:run
at makeError (/home/circleci/repo/node_modules/start-server-and-test/node_modules/execa/lib/error.js:56:11)
at handlePromise (/home/circleci/repo/node_modules/start-server-and-test/node_modules/execa/index.js:114:26)
at processTicksAndRejections (internal/process/task_queues.js:97:5) {
command: 'npm run cy:run',
exitCode: 1,
signal: undefined,
signalDescription: undefined,
stdout: undefined,
stderr: undefined,
failed: true,
timedOut: false,
isCanceled: false,
killed: false
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Exited with code exit status 1
CircleCI received exit code 1
complete log:
0 info it worked if it ends with ok
1 verbose cli [ '/usr/local/bin/node', '/usr/local/bin/npm', 'run', 'cy:run' ]
2 info using [email protected]
3 info using [email protected]
4 verbose run-script [ 'precy:run', 'cy:run', 'postcy:run' ]
5 info lifecycle [email protected]~precy:run: [email protected]
6 info lifecycle [email protected]~cy:run: [email protected]
7 verbose lifecycle [email protected]~cy:run: unsafe-perm in lifecycle true
8 verbose lifecycle [email protected]~cy:run: PATH: /usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/home/circleci/repo/node_modules/.bin:/usr/local/bin:/tmp/yarn--1587066858616-0.6618809648036963:/home/circleci/repo/node_modules/.bin:/home/circleci/.config/yarn/link/node_modules/.bin:/home/circleci/.yarn/bin:/usr/local/libexec/lib/node_modules/npm/bin/node-gyp-bin:/usr/local/lib/node_modules/npm/bin/node-gyp-bin:/usr/local/bin/node_modules/npm/bin/node-gyp-bin:/home/circleci/.local/bin:/home/circleci/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
9 verbose lifecycle [email protected]~cy:run: CWD: /home/circleci/repo
10 silly lifecycle [email protected]~cy:run: Args: [ '-c', '$(yarn bin)/cypress run' ]
11 silly lifecycle [email protected]~cy:run: Returned: code: 1 signal: null
12 info lifecycle [email protected]~cy:run: Failed to exec cy:run script
13 verbose stack Error: [email protected] cy:run: `$(yarn bin)/cypress run`
13 verbose stack Exit status 1
13 verbose stack at EventEmitter.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:332:16)
13 verbose stack at EventEmitter.emit (events.js:315:20)
13 verbose stack at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack at ChildProcess.emit (events.js:315:20)
13 verbose stack at maybeClose (internal/child_process.js:1026:16)
13 verbose stack at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)
14 verbose pkgid [email protected]
15 verbose cwd /home/circleci/repo
16 verbose Linux 4.15.0-1052-aws
17 verbose argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "cy:run"
18 verbose node v13.12.0
19 verbose npm v6.14.4
20 error code ELIFECYCLE
21 error errno 1
22 error [email protected] cy:run: `$(yarn bin)/cypress run`
22 error Exit status 1
23 error Failed at the [email protected] cy:run script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]
It seems that everything works fine until the cy:run script runs. I have no idea why it fails randomly. Any ideas?
> $(yarn bin)/cypress run
# leads to error...
command: 'npm run cy:run',
@AleksanderBodurri Since you mention yarn but the logs use npm did you try adding quotes to the command like in https://github.com/bahmutov/start-server-and-test/pull/256/files? I got warnings locally about the npm binary being off until I added the quotes and it switched to yarn...maybe I should just switch back to npm...
Hi all, i'm experiencing similar issues: In my case i'm running Storybook from a yarn workspace and all works fine locally but In Circle CI it doesn't seem to be observing the "wait"
I can see if i watch the task in Circle CI that "Run Cypress tests" is actually started before Storybook finishes, and depending on how long it takes for Storybook to start up sometimes means Cypress errors:
I'm not sure what this means or why it happens: HTTP(S) error for http://localhost:6006 Error: connect ECONNREFUSED 127.0.0.1:6006
Any help would be appreciated!
Thanks
package.json
"storybook": "yarn workspace docs storybook",
"cy:run": "cypress run",
"cy:pipeline": "start-server-and-test 'yarn storybook' http://localhost:6006 'yarn cy:run'"
config.yml
version: 2.1
orbs:
node: circleci/[email protected]
cypress: cypress-io/[email protected]
jobs:
run-build:
executor:
name: node/default
steps:
- checkout
- run: yarn install
- run: yarn build
run-unit-tests:
executor:
name: node/default
steps:
- checkout
- run: yarn install
- run: yarn test -u
workflows:
all-the-things:
jobs:
- run-build
- run-unit-tests
- cypress/install:
yarn: true
- cypress/run:
yarn: true
build: yarn build
requires:
- cypress/install
start: yarn cy:pipeline
logs
yarn run v1.22.4
$ ********************* 'yarn storybook' http://localhost:6006 'yarn cy:run'
********************* parsing CLI arguments: [ 'yarn storybook', 'http://localhost:6006', 'yarn cy:run' ] +0ms
********************* parsed args: { services: [ { start: 'yarn storybook', url: [Array] } ], test: 'yarn cy:run' } +6ms
1: starting server using command "yarn storybook"
and when url "http://localhost:6006" is responding with HTTP status code 200
running tests using command "yarn cy:run"
********************* single service "yarn storybook" to run and test +0ms
********************* starting server with command "yarn storybook", verbose mode? true +1ms
********************* starting waitOn http://localhost:6006 +6ms
********************* wait-on options { resources: [ 'http://localhost:6006' ], interval: 2000, window: 1000, timeout: 300000, verbose: true, strictSSL: true, log: true } +0ms
waiting for 1 resources: http://localhost:6006
making HTTP(S) head request to url:http://localhost:6006 ...
HTTP(S) error for http://localhost:6006 Error: connect ECONNREFUSED 127.0.0.1:6006
$ yarn workspace docs storybook
$ start-storybook --docs -p 6006 -s ./public --ci
info @storybook/react v6.0.21
info
making HTTP(S) head request to url:http://localhost:6006 ...
info => Loading static files from: /root/project/docs/public .
HTTP(S) error for http://localhost:6006 Error: connect ECONNREFUSED 127.0.0.1:6006
info => Loading presets
info => Loading presets
info => Loading config/preview file in "./.storybook".
info => Loading config/preview file in "./.storybook".
info => Adding stories defined in ".storybook/main.js".
info => Loading custom manager config.
info => Using default Webpack setup.
0% compiling
...
100%webpack built 446c4c3db0b389426589 in 9591ms
╭────────────────────────────────────────────────────╮
│ │
│ Storybook 6.0.21 started │
│ 8.7 s for manager and 11 s for preview │
│ │
│ Local: http://localhost:6006/ │
│ On your network: http://192.168.240.3:6006/ │
│ │
╰────────────────────────────────────────────────────╯
making HTTP(S) head request to url:http://localhost:6006 ...
HTTP(S) result for http://localhost:6006: { status: 200,
statusText: 'OK',
headers:
{ 'x-powered-by': 'Express',
'access-control-allow-origin': '*',
'access-control-allow-headers': 'Origin, X-Requested-With, Content-Type, Accept',
'accept-ranges': 'bytes',
'content-type': 'text/html; charset=UTF-8',
'content-length': '4466',
etag: 'W/"1172-v0SOHIZiw7G/ApxDDlPnHVfUjxs"',
date: 'Fri, 11 Sep 2020 21:04:45 GMT',
connection: 'close' },
data: '' }
wait-on(565) complete
********************* waitOn finished successfully +14s
********************* running test script command: yarn cy:run +1ms
$ cypress run
Errors
Cypress could not verify that this server is running:
> http://localhost:6006
We are verifying this server because it has been configured as your `baseUrl`.
Cypress automatically waits until your server is accessible before running tests.
We will try connecting to it 3 more times...
We will try connecting to it 2 more times...
We will try connecting to it 1 more time...
Cypress failed to verify that your server is running.
Please start this server and then run Cypress again.
Exited with code exit status 1
CircleCI received exit code 1
@PaulieScanlon
Try prefixing your server with http-get:// instead of http://
"cy:pipeline": "start-server-and-test 'yarn storybook' http-get://localhost:6006 'yarn cy:run'"
This was not the issue I was having but hopefully this helps.
I believe my issue is caused by flaky tests that for some reason do not display as actual failures when run with start-server-and-test.
I never had any tests fail locally until I started trying to run cypress multiple times in parallel. When running 3 cypress processes with headless chrome in parallel, some tests ended up being extremely flaky, usually failing in at least one of those runs.
I'm not 100% sure that the failing local tests are related to the flaky CI builds but it does seem odd. I'm still trying to figure out why those particular tests flake locally when cypress is run multiple times in parallel (they seem to never fail when only running cypress once).
@AleksanderBodurri Thanks so much for the response. I did see the http-get in the README and have run a few tests with it. I've asked in the Storybook Discord channel if anyone knows if Storybook uses webpack-dev-server but so far the tests seem to be just as flakey using http-get as they do without.
In both cases when the tests fail i see the same error message from wait-on
HTTP(S) error for http://localhost:6006 Error: connect ECONNREFUSED 127.0.0.1:6006
When i watch the Circle CI tasks both the Start task and Run Cypress tests task start at the same time which would suggest the wait-on part isn't quite working as expected.
I only have the one .spec at the moment and i'm not running multiple processes... just trying to work out where this issue is. Is it with start-server-and-test, Storybook, CircleCI or Cypress
I've opened an issue on the Storybook repo to see if that sheds any light on the matter.
Here's a short screen capture. The tests do pass but it's only because Storybook just about finishes loading before the last time Cypress attempts to run the test: start-sever-and-test-v1
Thanks again!
Not sure if this will help anyone but for my particular issue i had to understand a few key things. I'll start with my package.json
"storybook": "yarn workspace docs storybook",
"cy:test": "start-server-and-test 'yarn storybook' http-get://localhost:6006 'yarn cypress open'"
"devDependencies": {
"cypress": "^5.1.0",
"start-server-and-test": "^1.11.3"
}
When i run cy:test locally from my terminal everything does and has always worked fine. start-server-and-test does indeed wait for Storybook to start on 6006 before it runs cypress open
The problem was this wasn't observed when run through Circle CI
My Circle config.yml now looks like this:
- cypress/install:
name: cypress-install
yarn: true
- cypress/run:
name: start-storybook-and-test
yarn: true
requires:
- cypress-install
start: yarn storybook
wait-on: 'http://localhost:6006'
By adding wait-on to config.yml i'm able to correctly wait for Storybook to boot up and as the docs explain you can wait-for-server-to-respond and have the Cypress orb kick off the cypress run command without having to add anything additional to your package.json or config.yml because that's kinda what - cypress/run does for you.
So i guess the key takeaways here are there are the two different methods for using this.
- in the local environment where you do need use
start-sever-and-test - in production / Circle CI you don't as the
circleci/[email protected]orb does this for me.
Boogy time!
I'm not sure if I should open a new issue or tag onto this one but I've been having the this issue for a while now. Locally my tests run stable and never fail. On CircleCI it fails 100% of the time.
My CircleCI config
version: 2.1
jobs:
# The install job for all dependencies so we can reuse/depend on it in other jobs
install_dependencies:
docker:
- image: circleci/node:lts
steps:
- checkout
- restore_cache:
keys:
- v10-dependencies-{{ checksum "yarn.lock" }}{{ checksum "server/yarn.lock" }}
- v10-dependencies-
- run:
name: Install dependencies for app
command: yarn
- run:
name: Install dependencies for server
command: cd server && yarn
- run:
name: Adding debugging deps
command: sudo apt-get install tree # for debugging circle builds
- run:
name: Report file structure
command: tree -Ca
- run:
name: Node version
command: node -v
- run:
name: npm version
command: npm -v
- run:
name: Yarn version
command: yarn --version
- save_cache:
paths:
- .yarn
- .cache
key: v10-dependencies-{{ checksum "yarn.lock" }}{{ checksum "server/yarn.lock" }}
- persist_to_workspace:
root: ~/
paths:
- .yarn
- .cache
# Running all tests
testing:
docker:
- image: circleci/node:lts
steps:
- checkout
- attach_workspace:
at: ~/
- run:
name: Installing dependencies from cache
command: yarn
- run:
name: Installind dependencies for server from cache
command: cd server && yarn
- run:
name: Installing Cypress dependencies
command: sudo apt-get install xvfb libgtk-3-dev libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2
- run:
name: Testing our code
command: yarn test
- store_artifacts:
path: src/tests/utils/videos
- store_artifacts:
path: src/tests/utils/screenshots
workflows:
version: 2
running_jobs:
jobs:
- install_dependencies # First we install dependencies
- testing: # Second we test our code
requires:
- install_dependencies
The CI will fail when running the tasks
1: starting server using command "npm run test:e2e:server"
and when url "[ 'http://localhost:3000' ]" is responding with HTTP status code 200
running tests using command "npm run test:e2e:run"
See full run here: https://app.circleci.com/pipelines/github/dominikwilkowski/bronzies/129/workflows/24aa0111-6b86-4ac7-a607-0e5357528844/jobs/261
It will crash (?) the server via
Error: server closed unexpectedly
at ChildProcess.onClose (/home/circleci/project/node_modules/start-server-and-test/src/index.js:69:14)
at ChildProcess.emit (events.js:314:20)
at maybeClose (internal/child_process.js:1021:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)
I tried the different orbs and other things.
@dominikwilkowski I know your comment was long ago, but just for future developers reading this thread, I had a very similar issue and, in my case, upcon closer inspection I found and error ENOSPC: System limit for number of file watchers reached.
Then I successfully fixed the issue following the advice from this stack overflow answer
Sorry to revive after so long. I'm pretty sure I have a good idea of why this issue was occurring, at least in my case. I was writing tests that did not take full advantage of cypress' retry-ability mechanism. I was writing tests that looked like this:
cy.get('.foo').find('.bar').contains('baz')
Heres the scenario:
- A view exists like this
<div class="foo">
<div class="bar">
qux
</div>
</div>
- A cypress command causes some event that triggers the view to update. After the update, the view should look like
<div class="foo">
<div class="bar">
baz
</div>
</div>
Note that the divs after the update are not the same DOM elements as the divs before the update.
- Cypress then runs
cy.get('.foo').find('.bar').contains('baz'). If the page view doesn't change quickly enough (as is sometimes the case in CI containers where system resources are not as high as our local devices), thecy.get('.foo').find('.bar')part of the query matches against the elements from step 1. Since these elements will never contain'baz', even if thecontains('baz')is retried after the view updates,cy.get('.foo').find('.bar').contains('baz')will fail.
For some reason the details of the test failures were being silenced. Not sure exactly why that was happening, but I haven't seen the same flakiness since I refactored my tests to avoid the scenario above.
The biggest thing that helped me fix this issue was a snippet from the Cypress docs here https://docs.cypress.io/guides/core-concepts/retry-ability#Merging-queries
Specifically
Use cy.contains Tip: instead of cy.get(selector).should('contain', text) or cy.get(selector).contains(text) chain, we recommend using cy.contains(selector, text) which is retried automatically as a single command.
Documentation is once again undefeated 😄