Number Of Test Reported Is Incomplete

[Agent1997]

Describe the bug Number Of Test Reported Is Incomplete. Example Scenario: I have 2 Test classes with 4 test cases each. But on allure reports only 6 test are reported. NOTE: That the 2 missing tests are the first test cases on their respective test classes.

To Reproduce Steps to reproduce the behavior:

1. Execute a test on your IDE
2. Serve allure reports using allure serve
3. Compare the number of tests reported by TestNG and allure report
4. See that some tests are not reported by allure

Expected behavior All test executed regardless of their status should be reported correctly by allure.

Screenshots Allure-results

TestNG test summary

Allure Report

Environment (please complete the following information):

Allure version	2.13.8
Test framework	[email protected]
Allure adaptor	[email protected]

Additional context Test Codes

`public class SampleTest2 { private Object[][] data; private String wbloc = "src/test/resources/TestData/sample_request.xlsx"; private String sheetName = "API"; private int count = 0; @BeforeClass public void setData() throws IOException { data = XLSXReader.getNormalizedDataWithoutHeader(wbloc,sheetName); }

@Test(dataProvider = "data")
public void test(String request) throws JsonProcessingException {
    count++;
    AllureUtilities.setTestName(Integer.toString(count));
    SchemaRequestBuilder.create().request(request).build().run().validateSchemaAndStatusCode("schemas/createTestSuite.json",201).printResponse();
}

@DataProvider(name = "data")
public Object[][] getData(){
    return data;
}

}`

`public class SampleRequest { private Object[][] data; private String wbloc = "src/test/resources/TestData/sample_request.xlsx"; private String sheetName = "API"; private int count = 0; @BeforeClass public void setData() throws IOException { data = XLSXReader.getNormalizedDataWithoutHeader(wbloc,sheetName); }

@Test(dataProvider = "data")
public void test(String request) throws JsonProcessingException {
    count++;
    AllureUtilities.setTestName(Integer.toString(count));
    SchemaRequestBuilder.create().request(request).build().run().validateSchemaAndStatusCode("schemas/createTestSuite.json",201).printResponse();
}


@DataProvider(name = "data")
public Object[][] getData(){
    return data;
}

@AfterSuite
public void tearDown(){
    HashMap<String,String> env = new HashMap<>();
    env.put("Browser","Chrome");
    AllureUtilities.setTestEnvironment(env);
}

@BeforeSuite
public void setUp(){
    AllureUtilities.archiveAllureResults();
}

}`

POM.XML ` <groupId>org.example</groupId> <artifactId>API_Test_POC</artifactId> 1.0-SNAPSHOT

<name>API_Test_POC</name>

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <aspectj.version>1.8.10</aspectj.version>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
</properties>

<dependencies>
    <dependency>
        <groupId>org.testng</groupId>
        <artifactId>testng</artifactId>
        <version>6.14.3</version>
    </dependency>

    <dependency>
        <groupId>io.rest-assured</groupId>
        <artifactId>rest-assured</artifactId>
        <version>4.3.1</version>
    </dependency>

    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.11.2</version>
    </dependency>

    <dependency>
        <groupId>io.rest-assured</groupId>
        <artifactId>json-schema-validator</artifactId>
        <version>4.3.1</version>
    </dependency>

    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <version>1.18.12</version>
        <scope>provided</scope>
    </dependency>

    <dependency>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-testng</artifactId>
        <version>2.13.8</version>
    </dependency>

    <dependency>
        <groupId>ru.yandex.qatools.allure</groupId>
        <artifactId>allure-testng-adaptor</artifactId>
        <version>1.5.4</version>
    </dependency>

    <dependency>
        <groupId>org.apache.poi</groupId>
        <artifactId>poi</artifactId>
        <version>5.0.0</version>
    </dependency>

    <dependency>
        <groupId>org.apache.poi</groupId>
        <artifactId>poi-ooxml</artifactId>
        <version>5.0.0</version>
    </dependency>

    <dependency>
        <groupId>com.github.automatedowl</groupId>
        <artifactId>allure-environment-writer</artifactId>
        <version>1.0.0</version>
    </dependency>

</dependencies>

<build>
    <pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
        <plugins>
            <!-- clean lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#clean_Lifecycle -->
            <plugin>
                <artifactId>maven-clean-plugin</artifactId>
                <version>3.1.0</version>
            </plugin>
            <!-- default lifecycle, jar packaging: see https://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_jar_packaging -->
            <plugin>
                <artifactId>maven-resources-plugin</artifactId>
                <version>3.0.2</version>
            </plugin>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.0</version>
            </plugin>
            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.1</version>
                <configuration>
                    <argLine>
                        -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
                    </argLine>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>org.aspectj</groupId>
                        <artifactId>aspectjweaver</artifactId>
                        <version>${aspectj.version}</version>
                    </dependency>
                </dependencies>
            </plugin>
            <plugin>
                <artifactId>maven-jar-plugin</artifactId>
                <version>3.0.2</version>
            </plugin>
            <plugin>
                <artifactId>maven-install-plugin</artifactId>
                <version>2.5.2</version>
            </plugin>
            <plugin>
                <artifactId>maven-deploy-plugin</artifactId>
                <version>2.8.2</version>
            </plugin>
            <!-- site lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#site_Lifecycle -->
            <plugin>
                <artifactId>maven-site-plugin</artifactId>
                <version>3.7.1</version>
            </plugin>
            <plugin>
                <artifactId>maven-project-info-reports-plugin</artifactId>
                <version>3.0.0</version>
            </plugin>
        </plugins>
    </pluginManagement>
</build>

`

[Agent1997]

Additional note on the issue. 2 Test cases are identical on each of the test classes. Not sure if what I experience was a bug or a feature. Based on observation, if there are Identical test cases inside a test class, only 1 will be reported by allure. TestNG still execute those test cases even if they are identical so my expectation is that they should be reported by allure too. And if there are identical test classes inside a test suite defined under separate test tags, only 1 will be reported by allure. Please confirm if this is a feature. Thank you.

[NikhilPardeshi]

I am having the same issue. testng reported test-cases are different that allure reports. Allure reports are de-duping the similar test-cases

[amacostapagseguro]

I am having the same issue. I am using the Allure Report version 8.0.0. Are you found some solution?

[Agent1997]

@amacostapagseguro no confirmation from the devs yet if this a bug or not hence no solution yet. If I remember, my work around here was to make sure that test cases under a test class are unique and made sure that there were no same test classes under a test tag in your testng xml file. Hope this helps.

[EliaBahous-bwr]

I have the same issue @Agent1997, if you can go to the timeline you will see the other 2 tests... I don't know what to do with this issue !!

[ricardo7martins]

Same here, if I run allure serve ./allure-results, it shows all tests executed, but the html report generated with allure generate --clean has the first test executed always missing

[baev]

https://allurereport.org/docs/history-and-retries/