Is there any way to soft assert in MSTEST just like verify assert in TESTNG

[shirishnagarnz]

Description

I need to perform some UI sanity check under a single TestMethod. Out of 10 validation check in 1 TestMethod, if 3rd got failed, I want it to continue with 4th, 5th and so on instead of not executing rest of the code.

For a defect specific to the MSTest V2 test framework, describe the issue you've observed.

Steps to reproduce

Tried with try/catch block and without that but unable to perform soft assert in any way.

Expected behavior

Out of 10 validation check in a 1 TestMethod, if any inbetween validation fails, I want script continue to execute rest of the validation checks

Actual behavior

As any of the inbetween script/validation check fails, execution come out of TestMethod and not executing rest of the script

Environment

MSTest1.4

[shirishnagarnz]

Can I please have a reply on this issue?

[jayaranigarg]

@shirishnagarnz : Apologies for the delay. As of now, we don't have a way to soft assert in MSTest.

This is a new ask, so I will mark this as enhancement. We don't have plans to bring this feature in in near future, but we will certainly pick it up if we have enough votes. Thank you for you feedback!

[shirishnagarnz]

So any solution apart from soft assert?

I want all steps to get execute even though some in-between steps get fail.

[jayaranigarg]

@shirishnagarnz : Unfortunately there is no way to do so as of now.

[shirishnagarnz]

Thanks Jaya,

Don't know how many other users needs this enhancement but if it's not a big change, can your team please work on this enhancement?

[jayaranigarg]

Tagging @cltshivash @PBoraMSFT to look into the ask.

[shirishnagarnz]

Hi @jayaranigarg - any connecting to DB through selenium is available in MSTEST or MSTESTV2?

Please share some reference if so.

[jayaranigarg]

@shirishnagarnz : We would recommend you to open new issue for any more questions you have and not mix different issues on same thread. This is just to ensure that anybody looking for a resolution of the initial issue can search it easily and is not diverted with some other discussion on the thread.

[shirishnagarnz]

@shirishnagarnz : We would recommend you to open new issue for any more questions you have and not mix different issues on same thread. This is just to ensure that anybody looking for a resolution of the initial issue can search it easily and is not diverted with some other discussion on the thread.

Thanks @jayaranigarg . Logged as https://github.com/Microsoft/testfx/issues/576

[acesiddhu]

@shirishnagarnz as Jaya mentioned we dont have this support out of the box today in mstest framework. We are open to contributions on this one.

I also found this link https://automationrhapsody.com/soft-assertions-c-unit-testing-frameworks-mstest-nunit-xunit-net/ which uses FluentAssertions to implement soft assert. you can use that in your test.

[Grauenwolf]

My ideal proposal would be to allow an assert parameter.

[TestMethod] public void MyTest ( Assertions assert)

1. The assert parameter would be populated by the framework.
2. The Assertions class would expose all of the same methods as the Assert class.
3. Rather than throwing an exception, failures would be accumulated. By default, the framework would report the failures by checking the assert parameter after the function call ends.
4. Calling assert.Checkpoint() would immediately throw an exception if there are any captured failures. This would allow for existing the test early when there are known problems.

---

If this is too much, option B would be:

[TestMethod] public void MyTest ( Assertions assert)
{
     using var assert = new Assertions();
     
    //at the end of the function, assert's Dispose method would throw if there are any captured failures
}

Option B is not as good as option A because option A can capture a the regular exception along with the captured failures and report them together.

Option B will hide the regular exception because assert has to throw its own exception in the finally clause of the using blocks.

[Evangelink]

@shirishnagarnz @Grauenwolf I have also checked and confirm this is not available at the moment. I don't think it would be super complex to implement but we would need to design this correctly.

Out of curiosity, are you expecting to have some tests of a class to be "soft" and the other "hard" or would it be a good compromise for you to say all tests of a given test class would be doing "soft" assertions?

[Grauenwolf]

Honestly, I could say. This is something that we should have people try both ways to see which is better.

There is a risk of confusion if soft and hard asserts are mixed.

But if they can't be mixed, it may make some tests harder to write.

That said, we may not have a choice. It may turn out that the syntax dictates that both have to be offered.

[kboom]

I would strongly appreciate this feature. Without this simple feature, the quality of tests is much lower.

What would make the most sense for me is:

Assert.IsNotNull(x); // hard assertion, so that you know that x is not null, otherwise the test logic below would break
Assert.Softly(softAssert => {
   softAssert.AreEqual("x",x.A);
   softAssert.AreEqual("y",x.B);
});

But then the question is if we can somehow allow people to retain their existing custom assertions, so they can be still used.

I wouldn't worry about confusion of mixing assertions since the way above is explicit and it is also something that a lot of people are used to - done in most of the test frameworks.

[nohwnd]

I can see this working nicely with async local as well. it would be nice to allow soft assertions by setting that affects more than one test, and have a local opt-out.

Like

var requestResult = call server; // Hard assertion to guard my test from failing in 10 places when I did not even finish the request: Assert.AreEqual(200, requestResult.ReturnCode, throw: true);

soft assertions Assert.HaveCount(3, requestResult.Users); Assert.AreEqual("jakub", requestResult.Users[0].Name); etc.