Investigate test case association feature and its support with MTP

[Youssef1313]

https://learn.microsoft.com/en-us/azure/devops/test/associate-automated-test-with-test-case?view=azure-devops

It looks like this feature may be working by setting a VSTest property on TestCase, which means it may not work with MTP.

We need to investigate further.

[JoaoAntunes8210474]

@Youssef1313 Hello sir, hope you are doing well!

Would you be able to give any information or any plans/dates for this feature being supported by MTP?

[Youssef1313]

Hi @JoaoAntunes8210474

Unfortunately we didn't yet investigate what's needed to support this in MTP.

[JoaoAntunes8210474]

Hey @Youssef1313, hope you are doing well!

Would you be able to let me know if investigating how to support this feature in MTP is something that's planned for the near future?

It's a somewhat important feature for integrating tests with Azure DevOps' test cases.

[nohwnd]

I don't have the technical details, I've reached out to ADO team to see if they can point me to the code that does it.

[JoaoAntunes8210474]

@nohwnd Hello sir, hope you are doing well!

Would you happen to have any updates regarding contacting the ADO team for information over how this feature was previously supported on VSTest?

[nohwnd]

I got the supposed locations of the code, but did not get to investigate it yet, I will at best get to it next week.

[Youssef1313]

Related to https://github.com/microsoft/testfx/issues/5951

I assume the general flow that users do is:

• Associate test cases in Test Explorer to AzDO test case, and an AzDO test case is part of a test plan.
• Use VSTest AzDO task to run a given test plan by specifying testSelector: 'testPlan' and then specify the test plan they want to run.

What we need to do for MTP to support this:

1. Update Visual Studio to support associating test cases to AzDO test cases for MTP. We have a whitelisted set of adapters that we should include testingplatform-bridge in.
2. The NuGet package Microsoft.TeamFoundationServer.Client has APIs related to test plans. We need to figure out the exact API we need that exposes test cases in a given test plan (hopefully there will be something there in that package?).
3. We don't implement the support in an AzDO task. Instead, we create an AzDO test plan extension.
   • The test plan AzDO extension package will have a dependency on MS.TFS.Client
   • When the user specificies --azdo-test-plan, we will query the test plan, and use AddTestExecutionFilterFactory to construct a filter. But here, we will face an issue, as filters are not very well standardized, except for test node uids. I'm guessing that on VSTest, there was a special casing for MSTest/NUnit/xUnit when the filter is constructed.

[JoaoAntunes8210474]

@Youssef1313 The general flow you described is precisely the most common way we use the feature in our company.

We also have a dozen or so Azure Pipelines running tasks to request from the Azure DevOps API information on which test cases have the Automated status. Which is why this feature is very important to us (although we are probably not the only ones using the feature in this way).

[JoaoAntunes8210474]

@nohwnd Hello sir, hope you are doing well!

Would you be able to tell us if you've had the opportunity to investigate this issue?

[Youssef1313]

Unfortunately, we didn't yet prioritize an implementation for this feature.

[Youssef1313]

FYI @mckennabarlow

[elgatov]

This is preventing a very simple derived test method attribute from being associated with Azure Test Plans:

using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
using System.Collections.Generic;

[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class RetryingTestMethod : TestMethodAttribute
{
    private int retries;

    public RetryingTestMethod(string testName, int retries) : base(testName)
    {
        this.retries = retries;
    }

    public override TestResult[] Execute(ITestMethod testMethod)
    {
        var testResults = new List<TestResult>();
        do 
        {
            testResults.AddRange(base.Execute(testMethod));
        } while (--retries > 0 && !testResults.Exists(r => r.Outcome == UnitTestOutcome.Passed));

        return [.. testResults];
    }
}

[Youssef1313]

@elgatov Side note: We already have now RetryAttribute that you can use with [TestMethod].

For test case association though, this is so far not yet prioritized for Microsoft.Testing.Platform. Consider thumbing-up the issue so we have better idea how blocking this issue is for users migrating to Microsoft.Testing.Platform.

[shepherdm3]

@elgatov For test case association though, this is so far not yet prioritized for Microsoft.Testing.Platform. Consider thumbing-up the issue so we have better idea how blocking this issue is for users migrating to Microsoft.Testing.Platform.

Associate automated tests with test cases

Seriously! This feature has been present/working in the Professional and Enterprise versions of Visual Studio for at least the past 10+ years, and now the only means of associating test cases from AZDO with test methods in code is broken, having initially been reported nearly a year ago, and it has not yet been prioritized!?! (insert exasperated emoji)

[nohwnd]

Seriously! This feature has been present/working in the Professional and Enterprise versions of Visual Studio for at least the past 10+ years, and now the only means of associating test cases from AZDO with test methods in code is broken, having initially been reported nearly a year ago, and it has not yet been prioritized!?!

To not give other readers the wrong idea. The association does not work for MTP, it still works for VSTest, as it has been for the mentioned 10+ years. It is just that this feature has not been added to MTP yet.

If you are seeing the same problem with VSTest please report it there (microsoft/vstest), and we will fix it, as that is a regression in existing functionality rather than a "new" feature in the new platform.

[shepherdm3]

If you are seeing the same problem with VSTest please report it there (microsoft/vstest), and we will fix it, as that is a regression in existing functionality rather than a "new" feature in the new platform.

Brillant. So initially I did not understand the suggestion, but a bit more research and I find that in order to check if I observe the same issue with VSTest, I had to comment/remove the 'EnableMSTestRunner' line from the .csproj file for the tests. Having done that, I am once again able to see the dialog to associate test cases. The problem is that despite my co-worker seeing the full dialog, in my installation, the dialog is truncated and only a hint of the buttons to submit or cancel the action at the bottom of the dialog are visible.

Interesting to note that my coworker with Visual Studio Community see the full dialog:

[nohwnd]

Okay that looks like purely UI issue with the dialog. I just tried on latest Visual Studio, and I don't see the problem. I tried 4k resolution on 100% dpi, 4k on 125% dpi, and on 300% dpi.

And similarly with FullHD. 100%, 125%, 175%, still no repro.

What is you display resolution and dpi?

[nohwnd]

For future reference this is the view. And I found the definition for it.

[nohwnd]

@shepherdm3 Moved the UI issue here, https://github.com/microsoft/vstest/issues/15348 let's discuss there, to not pollute this issue with it.

[shepherdm3]

This issue can be closed.