PTVS Implement directory-based test containers

fixes https://github.com/microsoft/PTVS/issues/7043

The VS testing team has recently changed test discovery such that on machines with multiple CPU cores, it can discover tests from different sources in parallel inside different test host processes. We have been over discovering tests because our adapter would unnecessarily (re)discover tests in all the .py files in each process. So for example, if we have 3 sources and 2 tests files, our adapter is going to return 3 * 2 tests discovered in total.

The fix is to just supply a directory as the ITestContainer.Source from ITestContainerDiscoverer. The ITestDiscoverer will be called with the directory that we supply and it can discover tests in this directory when that happens.

Potential follow up changes that can be made in the future:

If we are discovering at the folder level (esp. for open folder where the folder is always the root folder), the incremental discovery could be slow. For example, if only one python test file has changed since the previous discovery then you would still be discovering all files under the folder.

Although this is no different than what we were doing before, it would be nice to somehow "remember" / persist the tests in .py files that haven't changed so that incremental discovery can be faster...

It could be a follow up change since the current switch to directory based would be no worse than what our adapter was already doing.

The only main difference from the previous test explorer is the way the tests are displayed in the Test Explorer's hierarchy. See more details here.

I ran some tests for different scenarios and it's working as expected. Test scenarios including project view, folder view, changing search path, adding/removing/modifying testing files.

Unittest, project view
Unittest, folder view
Pytest project view
Pytest, folder view

Aug 31 '22 01:08 StellaHuang95

The PR build will keep failing because you are using an attribute that was added to the most recent version of Visual Studio, and the agent pool that runs our builds has an older version. I've reached out to the DevDiv Fundamentals team to ask about a preview-pool or something similar. This PR will have to sit until they have a solution :)

Sep 15 '22 19:09 AdamYoblick

@StellaHuang95 The Test Platform APIs (including the above attribute) can also be consumed via NuGet packages. If you switch your projects to consume these via NuGet, you wouldn't need a matching version of VS installed in your CI for build - you would only need a matching VS version when you test the changes.

For example, the VS 17.4 version of MS.TP.OM.dll (which should contain the above attribute) should be available in this NuGet package https://www.nuget.org/packages/Microsoft.TestPlatform.ObjectModel/17.4.0-preview-20221003-03. Looks like other NuGet packages are already being referenced in this repo via this file - https://github.com/microsoft/PTVS/blob/main/Build/17.0/packages.config.

Tagging @nohwnd in case there are any other gotchas that we should be aware of regarding the Test Platform NuGet packages.