allure-java icon indicating copy to clipboard operation
allure-java copied to clipboard

Test results are not presented in the execution order same as TestNG does it.

Open OlegKuzovkov opened this issue 2 years ago • 2 comments

Describe the bug Our automated testing heavily depends on @BeforeSuite/@BoforeClass/@BeforeMethod methods. When we analyze test results it is crucial for us to see the execution order of tests in the report to quickly understand the sequence of events that cause a failure to happen. When we execute test cases from IDEA, we see test results for classes/beforeMethods/testMethods ordered by the execution: image

In Allure the same report will look completely different in terms of execution order: image

Even with this simple example this looks bad, but in our case the number of test cases is much bigger that makes our life very complicated when in comes to test result analysis.

To Reproduce Steps to reproduce the behavior:

  1. Use the code snippet below and execute test class with testNG:
package com.ctera.auto.tests;

import lombok.extern.log4j.Log4j2;
import org.testng.annotations.*;

import static io.qameta.allure.Allure.step;

@Log4j2
public class TestClass {
    @BeforeSuite
    public void beforeSuite() throws Exception {
       log.info("before suite");
    }
    @BeforeClass
    public void beforeClass() throws Exception {
       log.info("before class");
    }

    @BeforeMethod
    public void beforeMethod() throws Exception {
       log.info("before method");
        throw new Exception("Before Method failure");
    }

    @Test
    public void test1() {
        step("Step of the test", () -> {
        });
    }
    @Test
    public void test2() {
        step("Step of the test", () -> {
            log.info("in test");
        });
    }
    @Test
    public void test3() {
        step("Step of the test", () -> {
            log.info("in test");
        });
    }

    @AfterSuite
    public void afterSuite(){
        log.info("before suite");
    }
    @AfterClass
    public void afterClass(){
        log.info("after class");
    }
    @AfterMethod
    public void afterMethod(){
        log.info("after method");
    }

}
  1. Check results in IDEA
  2. Generate Allure Report and check results

Expected behavior Test cases should be organized in execution order, so we could easily detect which @BeforeMethod failed which test by just looking at the test results view, the same as TestNG does.

Screenshots Attached above.

Additional context Add any other context about the problem here.

OlegKuzovkov avatar Mar 06 '23 15:03 OlegKuzovkov

@OlegKuzovkov Good day! Could you provide more details please such as IDEA version / libraries versions I tried to reproduce but failed:

I am using:

1. TestNG - 7.7.0
2. allure-testng - 2.23.0
3. allure-java-commons - 2.23.0
4. IDEA - IntelliJ IDEA 2021.1.2 (Community Edition)
Build #IC-211.7442.40, built on June 1, 2021 (yes, I need to update)
image image

PavelMal avatar Aug 23 '23 14:08 PavelMal

Allure TestOps doesn't have a sort by execution order (aka start date).

baev avatar Aug 23 '23 19:08 baev