is any plan to support soft assertion?

[stevenxuwoss]

I'm submitting a ...

• [ ] bug report
• [1 ] feature request
• [ ] support request => Please do not submit support request here, see note at the top of this template.

What is the current behavior?

current step, which has failed soft assert in it, does not fail. The case fails when assertAll method is called.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

What is the expected behavior?

hope it show failed in each step and make the test failed

What is the motivation / use case for changing the behavior?

Please tell us about your environment:

• Allure version: 2.1.0
• Test framework: [email protected]
• Allure adaptor: [email protected]

Other information

[rad96]

@stevenxuwoss Can you provide some simple code to reproduce issue?

[lbastil]

There is a very old similar issue: https://github.com/allure-framework/allure-java/issues/19

[lbastil]

I would really appreciate an enhancement here. I guess using soft assertions, either from testng directly or via assertj or fest is not such an rare scenario. I tested both variants testng and assertj soft assertions and allure does not handle them in an reasonable way (like described above).

[lbastil]

On a very general level an example would be:

public class MyTestClass {

        private SoftAssert softAssert;
    
        @BeforeMethod(alwaysRun = true)
        public void initialize(Method method) {
    
            softAssert = new SoftAssert();
        }
    
        @Test
        public void myComplexTest_1() {
    
            reusableTestcasePart_1();
            reusableTestcasePart_2();
            softAssert.assertAll();
        }
    
        @Step("...")
        protected void reusableTestcasePart_1() {
            // do something directly ...
    
            // use reusable Steps:
            testStep_1();
            testStep_2();
        }
    
        @Step("...")
        protected void reusableTestcasePart_2() {
            // do something directly ...
    
            // use reusable Steps:
            testStep_3();
            testStep_4();
        }
    
        @Step("...")
        protected void testStep_1() {
    
            boolean stepResult = true;// in real case check some sophisticated ui part
            // soft assertion
            softAssert.assertTrue(stepResult);
        }
    
        @Step("...")
        protected void testStep_2() {
    
            boolean stepResult = false;// in real case check some sophisticated ui part
            // soft assertion
            softAssert.assertTrue(stepResult);
        }
    
        @Step("...")
        protected void testStep_3() {
    
            boolean stepResult = false;// in real case check some sophisticated ui part
            // soft assertion
            softAssert.assertTrue(stepResult);
        }
    
        @Step("...")
        protected void testStep_4() {
    
            boolean stepResult = true;// in real case check some sophisticated ui part
            // soft assertion
            softAssert.assertTrue(stepResult);
        }
    }

One would expect in this case the following report result:

• (test method) myComplexTest_1 : FAILED
  • (step) reusableTestcasePart_1 : FAILED
    • (step) testStep_1 : SUCCESS
    • (step) testStep_2 : FAILED
  • (step) reusableTestcasePart_2 : FAILED
    • (step) testStep_3 : FAILED
    • (step) testStep_4 : SUCCESS

But it is currently:

• (test method) myComplexTest_1 : FAILED
  • (step) reusableTestcasePart_1 : SUCCESS
    • (step) testStep_1 : SUCCESS
    • (step) testStep_2 : SUCCESS
  • (step) reusableTestcasePart_2 : SUCCESS
    • (step) testStep_3 : SUCCESS
    • (step) testStep_4 : SUCCESS

Which is misleading and hard to review for the test team which evaluates the allure report details

[vijaytabhatt]

any update regarding this? i am facing a similar issue

[skhomuti]

I'd suggest using https://github.com/okken/pytest-check. It works well with Allure

[esultanza]

I'd suggest using https://github.com/okken/pytest-check. It works well with Allure

Not pretty well: It marks failed step as pass

[skhomuti]

@esultanza, some scenarios probably don't work well. Could you provide code examples?

this example works as expected

from pytest_check import check

import allure


def test_something():
    with check, allure.step("Step 1"):
        assert 1 == 1
    with check, allure.step("Step 2"):
        assert 1 == 2
    with check, allure.step("Step 3"):
        assert 1 == 3

[esultanza]

@esultanza, some scenarios probably don't work well. Could you provide code examples?

this example works as expected

Your code works for me, thank you! My one included allure.step and check at the different context managers:

def test_1():
        with allure.step("2==2"):
            with check: assert 2==2
        with allure.step("2==3"):
            with check: assert 2==3

[skhomuti]

order is important here, it might be different context managers with correct order allure.step() decorator should catch an error and throw it up in the soft assert. In your case, soft assertion goes first, and allure step just didn't receive an error here