Failed test runs show as green in Slack

[you1anna]

Hi @YOU54F

On v1.1.8-alpha I am seeing failed runs resulting in green slack posts, as if they have passed.

We are only using SLACK_WEBHOOK_URL but I have seen this working previously, reflecting passed/failed runs correctly.

Is there a need to also supply SLACK_WEBHOOK_FAILED_URL to handle failed runs?

Thanks

[YOU54F]

You should be able to provide either SLACK_WEBHOOK_URL or SLACK_WEBHOOK_FAILED_URL to send failed alerts to a specific channel.

It should only returned a status of passed, if totalFailures === 0

Might be worth checking the test report json, that the logic is being used against

https://github.com/YOU54F/cypress-slack-reporter/blob/35d0f201e96b0b7200d562077d0074a609daa11b/src/slack/slack-alert.ts#L528

I can see value in debug mode, logging out the report contents to the command line,

[you1anna]

It's hard to debug this in CI because we don't control it. It would be good to understand how you are debugging this locally.

Are you just running script.ts with breakpoints, or can you suggest a better way? Thanks!

[YOU54F]

Just from looking at the code, and the points at which it determines the test status.

There are tests to cover the report status, that help locally. it would only return a pass if there is no failures in the test report.

I’ve got an free slack instance, os circleci account and a couple of projects that consume this that I use for remote testing, but I don’t use the custom url option, that was a feature request that I haven’t personally tested e2e.

One of the easiest ways to diagnose the test reports and artefact location is to ssh into the ci box if your ci provider supports that. I can do that on circleci to navigate around the box, check the various artefacts, cat the test report and check the contents.

Just read that you can’t control ci, yeah that’s a pain, I have the liberty of full access in non prod envs at work, but an open source account on circleci costs me nowt and was the easiest way for me to play around on this!

[YOU54F]

You could always add something into your ci script, a step to cat the test report and ls some of the artefact dirs, and store them for inspection later. 🤷‍♂️ Sorry I can’t be of much more help at the mo fella

[you1anna]

Hi @YOU54F we are still seeing instances of this happening in CI. I have an example of mochawesome.json that I could send to you, but would rather not share it here. You can see from the screenshot the build returns green but there is a failed.png file attached.

[YOU54F]

boo!

feel free to stick it in a private gist and you can send it to my email ( @.***) or email it directly. will take a look when I get some time, although I am busy on quite a few other projects atm.

On Mon, May 17, 2021 at 3:43 PM Robin Miklinski @.***> wrote:

Hi @YOU54F https://github.com/YOU54F we are still seeing instances of this happening in CI. I have an example of mochawesome.json that I could send to you, but would rather not share it here. You can see from the screenshot the build returns green but there is a failed.png file attached.

[image: Screenshot 2021-05-17 at 15 39 25] https://user-images.githubusercontent.com/7108767/118507788-77b62c80-b726-11eb-981d-6b79e498b375.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/YOU54F/cypress-slack-reporter/issues/995#issuecomment-842383403, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYCJ4PFRRBF77HPJFUXKUTTOETS5ANCNFSM43SNJ2DQ .

[you1anna]

Hey @YOU54F I've made the gist but unable to see the email you posted (*'d out). How can we solve this?