When tests fail during `github-action@v3` the step does not emit failure exit code

[cgraham-rs]

I'm executing parallelized tests recorded to the Cypress Dashboard using github-action@v3 Functionally everything works perfectly and as expected with the single exception that when tests fail the github-action@v3 step in my workflow appear with a grey checkmark instead of the expected red X. Thus that workflow run appears to have passed with a green checkmark when reviewing my Github workflow run history.

WORKS:

• The Cypress test are successfully executed in parallel
• Cypress successfully adds PR comments when the test execution has completed
• The correct pass/fail results are present in the PR comment
• The PR comment links to the correct Cypress Dashboard run
• github-action@v3 logs all have the correct pass/fail results
• github-action@v3 logs all link to the correct Cypress Dashboard run

DOES NOT WORK:

• github-action@v3 step looks like it passed even though it should have failed because there are failed tests

Workflow run showing cypress (chrome, 6) with a green checkmark, and github-action@v3 with a checkmark.

Expanding github-action@v3 and it clearly includes failed tests but does not include a failure exit for the step.

[admah]

@cgraham-rs thanks for filing this issue. There were recently some changes made in the Cypress Dashboard service addressing this (last week). Is this still an issue for you?

[cgraham-rs]

@admah Issue persists, just hit a bunch of test failures but the Github Action shows all green checkmarks. Let me know what I can do to help debug/troubleshoot.

[admah]

@cgraham-rs what version of Cypress are you using?

[cgraham-rs]

@admah

  │ Cypress:        10.4.0                                                                         │
  │ Browser:        Firefox 104 (headless)                                                         │
  │ Node Version:   v14.20.0 (/opt/hostedtoolcache/node/14.20.0/x64/bin/node)   

[admah]

@cgraham-rs thanks for that info. Is it possible for you to test this against the latest release - 10.7.0?

[admah]

I'm going to close this since it is apparently fixed with Cypress 10.7.0 and later. If it persists, please reopen.

[cgraham-rs]

@admah Sorry, took me a couple days to get back to this. It is still an issue on my end using Cypress 10.7.0. Let me know what additional debug I can provide.

To re-summarize:

• Github action/workflow shows green checkmark (no issues)
• All Cypress jobs in the action/workflow shows green checkmark (no issues)
• But one or more corresponding cypress-io/github-action@v3 within those jobs definitely have failed tests, and it ends with Done in 498.45s. instead of an error/failure code

[admah]

@cgraham-rs just circling back to this - is it possible for you to test against the latest version github-action@v4?

[cgraham-rs]

Confirmed this issue persists when using github-action@v4 which I updated to late last week.

@admah If you cannot reproduce, and nobody else is complaining about this (that I can see), I'm guessing it has something to do with my specific setup. Now that I'm rubber ducking via this comment... 🦆 🦆 🦆 .....we are executing Cypress via a custom script which in turn uses the Cypress Module API executing await cypress.run(argv) Ahhhhhh!!! But without any error handling per https://docs.cypress.io/guides/guides/module-api#Handling-errors

And as now that I've finally realized that and implemented the necessary error handling which clearly includes process.exit(result.totalFailed) I'm now getting actual exit codes from our custom script.

Before:

After:

This in turn should cause the Github action to finally see the non-0 exit code and flag our workflows in red as we've been expecting. I will confirm shortly.

[cgraham-rs]

@admah Sorry for the delay, I'm confirming the way we were using Cypress Module API was the root cause of my issues.

Unsure how to make this better, but some comments:

• The documentation exists and provides all of the code necessary to avoid this issue 🥇
• But I obviously didn't realize the impact in relation to my CI jobs
• Maybe call out how CI jobs require handling of results to correctly fail so people like me have less of a chance to fall into this scenario?

[admah]

@cgraham-rs thanks for the update, and I'm really glad that you were able to get to the root issue. Adding an extra callout about that is something we can do.

[kumarajiv1]

Hi @cgraham-rs , what solution worked for you? I am testing with Cypress 10.7.0 and using cypress-io/github-actions@v4 and even though my tests fails, Cypress job doesn't fail and in turn passes the github actions. Even gave module api a try and doing following `const cypress = require('cypress')

cypress.run() .then(result => { if (result.totalFailed){ process.exit(result.totalFailed) } }) .catch(err => { console.error(err) process.exit(1) })`

But, still no luck. As per @bahmutov 's PR, it should have failed once test failure is true. Not sure why it isn't happening. Any help will be appreciated.

[cgraham-rs]

@kumarajiv1 If you're just calling cypress:run directly I wouldn't expect you to have to do anything special.

If you're using the Cypress Module API you need to catch errors and report them back to the runner manually per the instructions. Code examples here: https://docs.cypress.io/guides/guides/module-api#Handling-errors

[kumarajiv1]

@cgraham-rs my first approach was to use cypress:run but then I noticed that even tests were failing, cypress job was still passing.

Then I came across your post and gave module API a try. As per the document

The Promise is only rejected if Cypress cannot run for some reason (for example if a binary has not been installed or it cannot find a module dependency). In that case, the Promise will be rejected with a detailed error.

And since an error is not produced, I am checking if the tests have failure, then I am exiting the process.

[kumarajiv1]

@cgraham-rs , appreciate it for making me take a look again. I was doing console.log instead of console.error. The solution worked for me. Thank you.

[kgartland-rstudio]

FYI, I came across this as well and found that if you're running Cypress within a Docker Container, the proper exit code may not be sent to GH Actions. To fix the issue, I added the docker compose option --exit-code-from to my docker compose up command.

[MikeMcC399]

@kgartland-rstudio

FYI, I came across this as well and found that if you're running Cypress within a Docker Container, the proper exit code may not be sent to GH Actions.

I couldn't reproduce the problem using the Cypress container browsers/node18.12.0-chrome106-ff106. This is one of the standard Docker containers posted to cypress-io/cypress-docker-images.

To fix the issue, I added the docker compose option --exit-code-from to my docker compose up command.

Are you using a Docker container in a different way to my example below?

I added this to the workflow .github/workflows/example-debug.yml so it looked like the following:

name: example-debug
on:
  push:
    branches:
      - 'master'
  pull_request:
  workflow_dispatch:

jobs:
  debug-logs:
    runs-on: ubuntu-22.04
    container: cypress/browsers:node18.12.0-chrome106-ff106
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Cypress debug logs 📝
        uses: ./
        env:
          DEBUG: '@cypress/github-action'
        with:
          working-directory: examples/basic

and I put a deliberate error into the examples/basic Cypress test.

When I ran the workflow it failed as expected:

You can find the run under 4136800503

[kgartland-rstudio]

I am using cypress/included:12.0.0.

I'm running this command from the Dockerfile after I install some packages I need on the box:

CMD cypress run --browser chrome --env api_key=${API_KEY}

The command I'm running in Github Actions is: docker compose up --exit-code-from cypress

[MikeMcC399]

@kgartland-rstudio

Thanks very much for adding your environment / steps. That makes it a lot clearer!