microsoft
Discuss whether `TestMethodAttribute.Execute` should return a single `TestResult` object
TestMethodAttribute.Execute currently returns TestResult[] but all known implementations return a single result.
This seems to currently be an API flaw that has been present only for historical reasons where previously parameterized tests where handled in DataTestMethodAttribute:
https://github.com/microsoft/testfx/blob/50c26d1cfcd364cfc604797bdcc7cede1a8f92e0/src/TestFramework/MSTest.Core/Attributes/DataTestMethodAttribute.cs#L25-L35
So, DataTestMethodAttribute had to return multiple test results. The design is now very different and the whole DataTestMethodAttribute isn't actually required now. So I feel like the API should change here and return a single TestResult for simplicity.
Still, we need to be very careful with changes here, in case there are valid use cases for returning multiple results (but I don't see that right now).
One valid use case for this might be executing tests against multiple endpoints or in different environments (like Sql Server and MongoDB) for quick A/B testing, like so
[AttributeUsage(AttributeTargets.Method)]
public sealed class SplitTestAttribute : TestMethodAttribute
{
public override TestResult[] Execute(ITestMethod testMethod)
{
ArgumentNullException.ThrowIfNull(testMethod);
// Run test with default endpoint (A)
TestClient.UseAlternativeUri = false;
var resultA = base.Execute(testMethod);
foreach (var r in resultA) r.DisplayName += " (A)";
// Run test with alternative endpoint (B)
var resultB = Array.Empty<TestResult>();
TaskExecutor.RunSynchronously(async () =>
{
TestClient.UseAlternativeUri = true; // That is an AsyncLocal
resultB = base.Execute(testMethod);
foreach (var r in resultB) r.DisplayName += " (B)";
});
return resultA.Concat(resultB).ToArray();
}
}
Obviously, this can be solved in other ways - probably those are even more cleanly or elegantly.
