Microsoft.TestPlatform (17.4.0-preview-20220726-02) not working

[alexanderjung-sdxag]

Description

Since updating to the latest version of Microsoft.TestPlatform, unittests don't run anymore in Azure DevOps pipelines. It seems as if there are some assembly reference mismatches, and test attributes are no longer recognized.

https://www.nuget.org/packages/Microsoft.TestPlatform 17.4.0-preview-20220726-02 <== error 17.2.0 (latestStable) <== works 17.4.0-preview-20220707-01 <== works

Steps to reproduce

Build pipeline step:

- task: VisualStudioTestPlatformInstaller@1
  displayName: 'Install Test Platform'

Expected behavior

Tests running fine.

Actual behavior

example 1:

  Failed TestFileArchiveOptions
  Error Message:
   Unable to set TestContext property for the class xxx.FileArchiveTest. Error: System.ArgumentException: Object of type 'Microsoft.VisualStudio.TestPlatform.MSTestFramework.TestContextImplementation' cannot be converted to type 'Microsoft.VisualStudio.TestTools.UnitTesting.TestContext'..
  Stack Trace:
      at System.RuntimeType.TryChangeType(Object value, Binder binder, CultureInfo culture, Boolean needsSpecialCast)
   at System.Reflection.MethodBase.CheckArguments(Object[] parameters, Binder binder, BindingFlags invokeAttr, CultureInfo culture, Signature sig)
   at System.Reflection.RuntimeMethodInfo.InvokeArgumentsCheck(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimePropertyInfo.SetValue(Object obj, Object value, Object[] index)
   at Microsoft.VisualStudio.TestPlatform.MSTestFramework.TestMethodRunner.SetTestContext()

The test class in question is fine:

    [TestClass]
    public class FileArchiveTest 
    {
        public TestContext TestContext { get; set; }

example 2:

  Failed RunActivity_SingleActivityWithBaggage_BaggageIsSet
  Error Message:
   Method xxx.ClassInitialize has wrong signature. The method must be static, public, does not return a value and should take a single parameter of type TestContext.

the method looks fine:

        [ClassInitialize]
        public static void ClassInitialize(TestContext context)

example 3:

  Failed TarjanTest5 [1 ms]
  Error Message:
   Test method xxx.TopoSortTest.TarjanTest5 threw exception: 
System.InvalidOperationException: Graph is not complete. Missing node 'v7' in adjacency list. Check if node is defined as key in the dictionary.

the ExpectedException attibute is not honored:

        [TestMethod]
        [ExpectedException(typeof(InvalidOperationException))]
        public void TarjanTest5()
        {

Diagnostic logs

failing output:

##[section]Starting: Install Test Platform
==============================================================================
Task         : Visual Studio test platform installer
Description  : Acquire the test platform from nuget.org or the tool cache. Satisfies the ‘vstest’ demand and can be used for running tests and collecting diagnostic data using the Visual Studio Test task.
Version      : 1.151.3
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/tool/vstest-platform-tool-installer
==============================================================================
Starting VsTest platform tools installer task.
==============================================================================
Looking for the latest pre-release version of the Microsoft.Testplatform.
[command]C:\CI\7\_work\_tasks\VisualStudioTestPlatformInstaller_2c65196a-54fd-4a02-9be8-d9d1837b7111\1.151.3\nuget.exe list packageid:Microsoft.TestPlatform -PreRelease -NonInteractive -Source https://api.nuget.org/v3/index.json
Microsoft.TestPlatform 17.4.0-preview-20220726-02
Found tool in cache: VsTest 17.4.0-preview-20220726-02 x64
VsTest will use the Test Platform package found in C:\CI\7\_work\_tool\VsTest\17.4.0-preview-20220726-02\x64
##[section]Finishing: Install Test Platform

working output:

##[section]Starting: Install Test Platform
==============================================================================
Task         : Visual Studio test platform installer
Description  : Acquire the test platform from nuget.org or the tool cache. Satisfies the ‘vstest’ demand and can be used for running tests and collecting diagnostic data using the Visual Studio Test task.
Version      : 1.151.3
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/tool/vstest-platform-tool-installer
==============================================================================
Starting VsTest platform tools installer task.
==============================================================================
Looking for the latest pre-release version of the Microsoft.Testplatform.
[command]C:\CI\1\_work\_tasks\VisualStudioTestPlatformInstaller_2c65196a-54fd-4a02-9be8-d9d1837b7111\1.151.3\nuget.exe list packageid:Microsoft.TestPlatform -PreRelease -NonInteractive -Source https://api.nuget.org/v3/index.json
Microsoft.TestPlatform 17.4.0-preview-20220707-01
Found tool in cache: VsTest 17.4.0-preview-20220707-01 x64
VsTest will use the Test Platform package found in C:\CI\1\_work\_tool\VsTest\17.4.0-preview-20220707-01\x64
##[section]Finishing: Install Test Platform

Environment

Azure Devops pipelines

Workarouds:

use earlier versions:

- task: VisualStudioTestPlatformInstaller@1
  displayName: 'Install Test Platform (latestStable!)'
  inputs:  
    packageFeedSelector: 'nugetOrg'
    versionSelector: 'latestStable'

or

- task: VisualStudioTestPlatformInstaller@1
  displayName: 'Install Test Platform (pinned version!)'
  inputs:
    packageFeedSelector: 'nugetOrg'
    versionSelector: 'specificVersion'
    testPlatformVersion: '17.4.0-preview-20220707-01'

[nohwnd]

Thanks for the report does this happen on any test project of just one specific test project?

[alexanderjung-sdxag]

Several different projects in our solution. Not all, but certainly not one specific one.

[greengumby]

This just occurred on our Azure Pipelines. Only thing to change is the version

Microsoft (R) Test Execution Command Line Tool Version 17.4.0 (x64)

[MarcoRossignoli]

Does it fail again on 17.4.0? Which version of MSTest are you using?

[brandscill]

We have the same problem. Be good to know if there is any solution.

[nohwnd]

@brandscill which exact version are you having issues with please?

[brandscill]

@nohwnd 17.4.1

[GoAvalon]

Started facing something similar. Tests were running fine till v17.3.2 But in v17.5.0 (which our build now uses), the tests are failing with

Error Message: Method Microsoft.XYZ.Tests.TestEnvironment.AssemblyInitialize has wrong signature. The method must be static, public, does not return a value and should take a single parameter of type TestContext.

[nohwnd]

@GoAvalon Can you double check your Microsoft.XYZ.Tests.TestEnvironment.AssemblyInitialize method, does it meet what is described in the error? What version of testing framework are you using?

[GoAvalon]

@nohwnd Yes, our AssemblyInitialize method already satisfies everything mentioned in the error. We are using Microsoft.NET.Test.Sdk version 16.5.0 Locally all tests run fine on the latest Visual Studio 2022 If I manually install vstest 17.3 then our tests run in the build pipeline too.

Only with vstest 17.5 in our build pipelines we see this error that I mentioned above.

[nohwnd]

@GoAvalon do you use mstest or xunit or nunit? Which version? :)

[GoAvalon]

@nohwnd We are using MSTest version 2.1.1

[nohwnd]

I am doing this, and it works (2.1.1. was not working with 17.3.0, but 17.5.0 works).

<!-- file mstest68.csproj -->
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>

    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.5.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.1.1" />
    <PackageReference Include="MSTest.TestFramework" Version="2.1.1" />
    <PackageReference Include="coverlet.collector" Version="3.1.2" />
  </ItemGroup>

</Project>

// file UnitTest1.cs
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;

namespace mstest68
{

    [TestClass]
    public class UnitTest1
    {
        [AssemblyInitialize]
        public static void AssemblyInitialize(TestContext context)
        {
            Console.WriteLine("AssemblyInitialize");
        }

        [TestMethod]
        public void TestMethod1()
        {
        }
    }
}

Any suggestions how to repro?

[GoAvalon]

@nohwnd I was trying to repro this but ended up solving the issue in my code. What was happening is that one of our test projects referenced another test project. When VSTest ran, it discovered the referenced project multiple times, and the first time the test ran it was successful, but when it ran again (due to it being detected twice), it failed with the above AssemblyInitialize error.

To fix this I removed the test project reference and now vstest step succeeds. Something probably changed in VSTest from 17.3 to 17.5 because we had this project reference since a long time, but at the very least my issue is resolved by removing all extra references.

[Evangelink]

Hey there,

What's the status here? Is there still some issue going-on?