allure2 icon indicating copy to clipboard operation
allure2 copied to clipboard

Published 20 hours ago • verified publisher icon allure-framework

Using @flaky or @muted or @known tags in Cucumber Scenarios or Feature doesn't reflect in Allure report

Open gnazgit opened this issue 2 years ago • 10 comments

Describe the bug Using @flaky or @muted or @known tags in Cucumber Scenarios or Feature doesn't reflect in Allure report. No markers were present in the report when using these tags.

To Reproduce Steps to reproduce the behavior:

  1. Create any cucumber feature file
  2. Add flaky or muted or known tags to either scenario or feature
  3. Observe the report markers

Expected behavior When marked @flaky, scenario or feature should be marked as flaky and should able to filter using Markers in the allure report.

Screenshots Please refer screenshot for expected behavior.

Environment (please complete the following information):

Allure version 2.17.3
Test framework [email protected]
Test framework cucumber@7
Allure adaptor [email protected]

With flaky symbol

@baev - please suggest any possible solution. Thanks!

gnazgit avatar Mar 11 '22 11:03 gnazgit

Hi @baev we are also facing the same problem, please suggest

dineshmcac avatar Mar 11 '22 11:03 dineshmcac

Hello @baev I'm also facing this issue with Cucumber6: allure-cucumber6-jvm

MSlaski avatar Apr 05 '22 10:04 MSlaski

Hello @baev Do you have any plan to fix this issue?

MSlaski avatar May 23 '22 19:05 MSlaski

@gnazgit @dineshmcac @baev

I found a workaround for this issue related to @flaky annotation and I will prepare small script to fix report (don't have time to find a root cause at allure's source code)

Basically after test inside allure-results directory you can find some files, the files which we have to modify are <TEST_ID>-result.json

If test will fail inside a root part of JSON there will be a structure called detailsStatus (sth like that), there all annotations are set to FALSE, but every step has also similar structure (JSON root > steps) and there you can find that, flaky value will be set top true if you added annotation to feature or scenario. Looks like the issue is to aggregate flaky status from steps and set proper value to overall flaky status

My script will be doing 4 things:

  1. Iterate over all *-result.json files (1 test will generate one result.json file)
  2. Check test status - if failed go further
  3. Check flaky status inside a first step - if set to true go further
  4. change value of overall flaky status

I will trigger this script at Jenkins stage before it will generate report

Manually all is working fine with this approach

MSlaski avatar May 26 '22 10:05 MSlaski

Hi, Is there any solution for this issue as of today? I am running the tests with following dependecies

  • Selenium Java 4.6
  • Cucumber-TestNG 7.9
  • TestNG 7.9
  • allure-cucumber7-jvm 2.20.1

The Test Report does not properly mark the scenarios that are tagged with @flaky

itkhanz avatar Dec 28 '22 22:12 itkhanz

Can someone help me out with the same issue? Allue 2.21.0, Cucumber 7, Junit 5.

Phlegethonyarre avatar Feb 18 '23 08:02 Phlegethonyarre

Hi, the same issue occurs in allure-cucumber5-jvm.

mdaraz avatar Apr 13 '23 07:04 mdaraz

Still happening on allure-cucumber7-jvm 2.21.0. Hi @mpkorstanje there is any standard workaround for this? commenting to keep this issue open.

sanflg avatar May 24 '23 12:05 sanflg

I think that the maintainers removed the support for the @Flaky annotation without updating the docs.

noomorph avatar Jul 28 '23 07:07 noomorph

We're looking for a solution too.

Allure 2.24.0. @badeball/cypress-cucumber-preprocessor 15.0.0. Cypress 12.14.0

dirkrooijakkers avatar Oct 04 '23 06:10 dirkrooijakkers