Add support for soft assertions

[baev]

[scripnichenko]

Hi @baev. Any plans to get it implemented? :bulb:

[baev]

@scripnichenko we are not using soft assertions in our code and you are the first one who are asking for such issue during last few years. Seems like everyone uses AssertJ instead of TestNG assertions this days. But I'm understand that issue is important for people who support old projects written years ago and have no possibility for refactoring.

BTW I have no idea how to fix this, some research work is required. Probably the easiest way to fix the issue add an aspect. If you have any ideas feel free to share.

[Little-Elephant]

I'll be the second one.

[Lyisko]

I'll be the third one.

[vlayon]

Hello there. I want to ask also for implementing SoftAsserts in Allure. Are you going to do it?

..because now the bahavior is : current step, which has failed soft assert in it, does not fail. The case fails when assertAll method is called.

[bereg2k]

Yep, that's an important feature.

[Guter1988]

I use it too. And due to this, add attachments when it failed to find a better solution.

[sskorol]

Can anyone explain the expected behavior and attach the minimal code to reproduce the issue? As I don't really understand what kind of SA support is required.

[Guter1988]

I will try from my point of usage: When in the an of API (for example), I'm checking the JSON response, I need to test all fields that exist as expected. I can run soft assertion or run a new test to validate one filed. Or in UI - when by click on an element will be an open window with 3 parameters that I need to validate, I will use SA.

I hope it helped :)

[sskorol]

@guterscooter I know how SA works. I wonder what do you expect to see in Allure while using SA?

[Guter1988]

@sskorol To see all parts where it failed. I see only the first one, while the next can failed will not be shown at Allure.

[twixdouble]

@sskorol mark each failed step (and each of its ancestors) with red color, add assertion message within failed step content, add stack-trace within assertion error content, add summary error-message as test result, e.g. 'N assertions failed: <error 1> (expandable with stack-trace), <error 2> (expandable) ...

[lbastil]

@sskorol: I think in #529 everything is explained quite well:

What is the current behavior? current step, which has failed soft assert in it, does not fail. The case fails when assertAll method is called.

What is the expected behavior? hope it show failed in each step and make the test failed

Example + Expectation:

On a very general level an example would be:

public class MyTestClass {

        private SoftAssert softAssert;
    
        @BeforeMethod(alwaysRun = true)
        public void initialize(Method method) {
    
            softAssert = new SoftAssert();
        }
    
        @Test
        public void myComplexTest_1() {
    
            reusableTestcasePart_1();
            reusableTestcasePart_2();
            softAssert.assertAll();
        }
    
        @Step("...")
        protected void reusableTestcasePart_1() {
            // do something directly ...
    
            // use reusable Steps:
            testStep_1();
            testStep_2();
        }
    
        @Step("...")
        protected void reusableTestcasePart_2() {
            // do something directly ...
    
            // use reusable Steps:
            testStep_3();
            testStep_4();
        }
    
        @Step("...")
        protected void testStep_1() {
    
            boolean stepResult = true;// in real case check some sophisticated ui part
            // soft assertion
            softAssert.assertTrue(stepResult);
        }
    
        @Step("...")
        protected void testStep_2() {
    
            boolean stepResult = false;// in real case check some sophisticated ui part
            // soft assertion
            softAssert.assertTrue(stepResult);
        }
    
        @Step("...")
        protected void testStep_3() {
    
            boolean stepResult = false;// in real case check some sophisticated ui part
            // soft assertion
            softAssert.assertTrue(stepResult);
        }
    
        @Step("...")
        protected void testStep_4() {
    
            boolean stepResult = true;// in real case check some sophisticated ui part
            // soft assertion
            softAssert.assertTrue(stepResult);
        }
    }

One would expect in this case the following report result:

• (test method) myComplexTest_1 : FAILED
  • (step) reusableTestcasePart_1 : FAILED
    • (step) testStep_1 : SUCCESS
    • (step) testStep_2 : FAILED
  • (step) reusableTestcasePart_2 : FAILED
    • (step) testStep_3 : FAILED
    • (step) testStep_4 : SUCCESS

But it is currently:

• (test method) myComplexTest_1 : FAILED
  • (step) reusableTestcasePart_1 : SUCCESS
    • (step) testStep_1 : SUCCESS
    • (step) testStep_2 : SUCCESS
  • (step) reusableTestcasePart_2 : SUCCESS
    • (step) testStep_3 : SUCCESS
    • (step) testStep_4 : SUCCESS

Which is misleading and hard to review for the test team which evaluates the allure report details

[lbastil]

@sskorol : if I can I would try to support the feature enhancement. Can you give me some advise about entry points where to start? I guess there is a kind of post-processing for the final test / step state (updating the TestResult information).

[deisenberg-remedy]

What is the status on implementing Soft Assertion support as described above?

[Phlegethonyarre]

Hi guys. A little workaround here I created long time ago, this could be helpful for your own ideas, probably with aspects :)

public class StepUtil {

	private final List<Throwable> troubles = new ArrayList<>();
	private final List<String> causes = new ArrayList<>();

	public void runStep(Object name, Runnable runnable) {
		final String uuid = UUID.randomUUID().toString();
		final List<String> localCauses = new ArrayList<>();
		final List<Throwable> localExceptions = new ArrayList<>();

		try {
			Allure.getLifecycle().startStep(uuid, new StepResult().setName(name.toString()));
			runnable.run();
		} catch (Throwable e) {
			localCauses.add(String.format("\r\nFailed step name: %s", name.toString()));
			localExceptions.add(e);
		}

		if (!localExceptions.isEmpty()) {
			Allure.getLifecycle().updateStep(uuid, (step) -> step.setStatus(Status.FAILED));
			troubles.addAll(localExceptions);
			causes.addAll(localCauses);
		} else {
			Allure.getLifecycle().updateStep(uuid, (step) -> step.setStatus(Status.PASSED));
		}

		Allure.getLifecycle().updateStep(step -> {
			List<StepResult> stepResults = step.getSteps();
			if (stepResults.stream().anyMatch(stepResult -> stepResult.getStatus().equals(Status.FAILED))) {
				step.setStatus(Status.FAILED);
			} else if (stepResults.stream().anyMatch(stepResult -> stepResult.getStatus().equals(Status.BROKEN))) {
				step.setStatus(Status.BROKEN);
			}
		});

		Allure.getLifecycle().stopStep(uuid);
	}

	public void checkTroubles() {
		if (!troubles.isEmpty()) {
			throw new AssertJMultipleFailuresError(causes.toString(), troubles);
		}
	}

[Achitheus]

Hi guys. A little workaround here I created long time ago, this could be helpful for your own ideas, probably with aspects :)

public class StepUtil {

	private final List<Throwable> troubles = new ArrayList<>();
	private final List<String> causes = new ArrayList<>();

	public void runStep(Object name, Runnable runnable) {
		final String uuid = UUID.randomUUID().toString();
		final List<String> localCauses = new ArrayList<>();
		final List<Throwable> localExceptions = new ArrayList<>();

		try {
			Allure.getLifecycle().startStep(uuid, new StepResult().setName(name.toString()));
			runnable.run();
		} catch (Throwable e) {
			localCauses.add(String.format("\r\nFailed step name: %s", name.toString()));
			localExceptions.add(e);
		}

		if (!localExceptions.isEmpty()) {
			Allure.getLifecycle().updateStep(uuid, (step) -> step.setStatus(Status.FAILED));
			troubles.addAll(localExceptions);
			causes.addAll(localCauses);
		} else {
			Allure.getLifecycle().updateStep(uuid, (step) -> step.setStatus(Status.PASSED));
		}

		Allure.getLifecycle().updateStep(step -> {
			List<StepResult> stepResults = step.getSteps();
			if (stepResults.stream().anyMatch(stepResult -> stepResult.getStatus().equals(Status.FAILED))) {
				step.setStatus(Status.FAILED);
			} else if (stepResults.stream().anyMatch(stepResult -> stepResult.getStatus().equals(Status.BROKEN))) {
				step.setStatus(Status.BROKEN);
			}
		});

		Allure.getLifecycle().stopStep(uuid);
	}

	public void checkTroubles() {
		if (!troubles.isEmpty()) {
			throw new AssertJMultipleFailuresError(causes.toString(), troubles);
		}
	}

Maybe I'm wrong, but it looks like the run() method never throws errors because it has a soft assertion inside?

[Achitheus]

I did this for my needs:

public class AllureSelenideCustom {
    public static void markOuterStepAsFailedAndStop() {
        AllureLifecycle lifecycle = getLifecycle();
        lifecycle.updateStep(step -> step.setStatus(Status.FAILED));
        lifecycle.stopStep();
    }

    public static void stepSoftAssert(final String passMessage, final String failMessage,
                                      final Consumer<String> consumer, final boolean markStepAsFailed) {
        Allure.step(markStepAsFailed ? failMessage : passMessage
                , () -> {
                    consumer.accept(failMessage);
                    if (markStepAsFailed) {
                        markOuterStepAsFailedAndStop();
                    }
                });
    }
}

The bad new is you should pass to this method boolean markStepAsFailed. It means you should additionally ("manually") check result before use it. Its ok for me in cases when I looking for "bad items" and pass as argument something like badItem.exists()