[Feature]: Add differ extensibility for MSTest assertions

[Youssef1313]

Summary

One current issue for assertions is the level of clarity of error messages. There is probably no one solution that fits all scenarios.

Background and Motivation

We should aim for a good experience when a test fails. For that, a degree of extensibility might be best.

Proposed Feature

public sealed class Assert
{
+    public void AddExtension(IAssertDiffExtension assertDiffExtension);
}

+ public interface IAssertDiffExtension
+ {
+     // TODO: Should this take the original objects instead?
+     bool IsSupported(Type typeOfExpected, Type typeOfActual);
+     string Diff(object expected, object actual);
+ }

Multiple extensions can be added during AssemblyInitialize. When an assert fails and wants to diff two objects, we will loop over all registered diff extensions (in reverse order). The first found extension that returns true from IsSupported will be used (which is the last registered because we loop in reverse order - more on the motivation of reverse ordering later in the proposal).

• This allows differs to be implemented outside of MSTest.TestFramework, as separate packages. There can be some provided by us, and others provided by the community.
  • One example is using DiffPlex here, potentially with different options that users can configure. For example, users who knows they have lots of multi-line string comparisons can install a differ that plays more nice with that, etc.
• Having the extensibility here allows TestFramework to not require additional dependencies (e.g, DiffPlex). The additional dependency is installed only when user explicitly wants the DiffPlex differ extension.
• The actual storage of the extensions can be an async local.
  • When installing in AssemblyInitialize, it makes it available to all tests.
  • Two different test classes can be running in parallel, and each test class could install a different differ because of the nature of the tests in those specific test classes.
  • That is why the "last registered" wins rule. If I install a differ in AssemblyInitialize (registered first), I want it to apply it everywhere. If later I installed one in ClassInitialize, I want that to be "more specific" and win.

[nohwnd]

I think this needs a lot of design. For example the textual diff feels like it should be built in, without any user action. It also feels like in cases where we don't write to console we don't need that diff, and can offload to external tools that do it on-demand (like VSCode).

I also don't see when users would want to register this kind of extension, rather than providing their own set of assertions, maybe showing diff is a smaller task to implement than a whole test suite, but if those two become disconnected, then I think we will have hard time getting consistent output that makes sense.

To me presenting data in assertion failures in a way that makes sense is one of the hardest problems a test framework needs to tackle, and allowing the data to be shown in any way the user wants makes it even harder.

But I'd be happy to brainstorm on this with you (not today though :D)