Can I add "post actions" after (null).Should().BeNotNull() is triggered or any other FluentAssertions' extensions ?

[cbries]

Description

I use fluentassertions in UI tests, works great. Do you provide any approach for adding callbacks in case FluentAssertions' extensions are triggerd?

For example, when I use the following snippet:

[..snip..]
            var guid = GetGuidOf(srvExt0);
            var resWait = WaitFor((out bool b) => b = refMgr.IsReferenceAdded(guid));
            resWait.Should().BeTrue();
[..snip..]

In case resWait is false I like to create a screenshot of the UI.

Can I register callbacks for this case somehow? Globally would be great to keep the tests clean and simple.

[dennisdoomen]

Unfortunately, no. There's an AssertionScope that takes an IAssertionStrategy that you could potentially use for this, but it's constructor is private. We would be open to a PR to change that though.

[krajek]

@cbries could you describe your use case in detail, please? Do you want to register some code to be run for any failed test? Are you sure that test frameworks themselves do not have this kind of callbacks built-in?

[eNeRGy164]

I guess he wants something like: resWait.Should().BeTrue().Otherwise(() => webDriver.TakeScreenshot());

[dennisdoomen]

Or a generic hook that allows you to run some code before and after the IAssertionStrategy

[cbries]

@krajek Sure the unittest framework some support that case (somehow). But between the exception and the finalization of the unit test our UI can change completly -- because time is crucial and context changes within the UI are very important and most of our cases (when issues are reported by our customers) depend on the UI state and which toolwindows or editors are open. By the way... how do I know which unit tests failed? Currently I do not know any way to figure out which method failed to create inidivudal screenshot names... By using the callstack?

>	TcHmiCore_Tests.dll!TcHmiCore_Tests.TestsVersion.Cleanup() Line 13	C#
 	[Native to Managed Transition]	
 	[Managed to Native Transition]	
	Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter.dll!Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestExecuter.RunCleanupMethod()	Unknown
 Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter.dll!Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestExecuter.ExecuteTest()	Unknown
	Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter.dll!Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestExecuter.Execute(Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestResult result)	Unknown
	Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter.dll!Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestRunner.ExecuteSingleTest(Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestExecuter executer, Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestElement test, Microsoft.VisualStudio.TestTools.Execution.ITestContext testContext, Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestAdapterContext userContext, bool isLoadTest)	Unknown
	Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter.dll!Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestRunner.Run(Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestElement test, Microsoft.VisualStudio.TestTools.Execution.ITestContext testContext, bool isLoadTest, bool useMultipleCpus)	Unknown
 	[AppDomain (QTAgent32_40.exe, #1) -> AppDomain (TestAppDomain: 837c96c9-6d62-40ca-b26e-4626b9fff38f, #3)]	
	Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter.dll!Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestAdapter.Run(Microsoft.VisualStudio.TestTools.Common.ITestElement testElement, Microsoft.VisualStudio.TestTools.Execution.ITestContext testContext, bool isLoadTest)	Unknown
	Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter.dll!Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestAdapter.Run(Microsoft.VisualStudio.TestTools.Common.ITestElement testElement, Microsoft.VisualStudio.TestTools.Execution.ITestContext testContext)	Unknown
	Microsoft.VisualStudio.QualityTools.AgentObject.dll!Microsoft.VisualStudio.TestTools.Agent.AgentExecution.CallAdapterRunMethod(object obj)	Unknown
 	mscorlib.dll!System.Threading.ThreadHelper.ThreadStart_Context(object state)	Unknown
 	mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)	Unknown
 	mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)	Unknown
 	mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)	Unknown
 	mscorlib.dll!System.Threading.ThreadHelper.ThreadStart(object obj)	Unknown

I have to notice that I am mainly developing VisualStudio 2013/2015/2017 extensions, no web stuff, so our testadapter does start VS-instances, queries for magic dte instances (see https://docs.microsoft.com/en-us/dotnet/api/envdte?redirectedfrom=MSDN&view=visualstudiosdk-2017) and runs our tests.

This is the product: https://www.beckhoff.com/TwinCAT-HMI/

The mentioned approach by @eNeRGy164 looks interesting. In that case I could make some fancy setups for the screenshots (e.g. where to store, naming, how many screenshots, probably 10 times after exceptions to see if somethings changes within the ui).

The other way looks good enough as well, mentioned by @dennisdoomen, but it should be possible to set these hooks in any unit test individually. Not only during testclass-initialize, it should be possible to set them up at any time.

In case my functionality-wish in this issue makes no sense, just tell me... I appreciate any comment. One of our testcases is as follows:

[TestMethod]
[TestCategory("Designer")]
[HostType("VSTestHost")]
        public void AutomoveCaret()
        {
            var hmiPrj = CreateProjectAndWaitForDefaultEditor("AutomoveCaret") as TcHmiProjectNode;

            var pane = hmiPrj.QueryPane("Desktop.view");
            pane.Should().NotBeNull();
            var splitPanel = pane.SplitPanel;
            splitPanel.Should().NotBeNull();

            if (!splitPanel.IsSplitted)
                splitPanel.ToggleFullscreen();

            WaitFor((out bool b) => b = splitPanel.IsSplitted);
            TcHmiVsUi.TimingUtils.Wait1();

            var ctrlid = "ViewDesktopBeckhoffLogo";

            var textlines = pane.HtmlSource;
            textlines.Should().NotBeNull();
            textlines.PlaceCaret(ctrlid);

            bool resWait = WaitFor((out bool b) => b = hmiPrj.IdentifiersSelected.Count > 0 && hmiPrj.IdentifiersSelected[0].Equals(ctrlid));
            resWait.Should().BeTrue("after caret move the identifier should be: " + ctrlid);

            splitPanel.ToggleFullscreen();

            VsWindow.GetInnerRect().CreateScreenshot(GetMethodName());
        }

At the end you can see how I do some screenshot of the current state when the test were succesfully. I like to call something like this at any time when Should().BeXYZ()failed or probably when it were run sucessfully as well. VsWindow.GetInnerRect().CreateScreenshot(GetMethodName());

[dennisdoomen]

Alternatively, you can register a central handler by setting the Services.ThrowException property.

[Evangelink]

@dennisdoomen I said in my PR that this thread was only partially fixed but after a double reading I tend to think this is actually totally fixed. WDYT?

[dennisdoomen]

Yes. They can wrap their call in an using new AssertionScope() that takes a custom IAssertionStrategy. I think that should be enough.