testfx
testfx copied to clipboard
microsoft
Incorrect test result view in Azure DevOps after upgrading MSTest to 2.2.4 and higher
Description
After I updated MSTest.Framework and MSTest.Adapter to 2.2.4 and higher, the test results in Azure DevOps are displayed incorrectly. Each test interation is displayed in a new line. Before that, the interations were shown under a test case.
For example: A Test with 2 iterations
Before: TestCase12
- Iteration1
- Iteration2
Currently
- TestCase12
- TestCase12
This a screenshot for the test results on Azure DevOps
For example: Test Case 1129
TRX-File look like different
After the upgrade, the trx file does not contain an <InnerResults> element
Hier is a screenshot fom trx-file
I use VS 2019 16.11.6 I'm not sure if this is a bug or a new configuration. This is my test configuration: Configure unit tests by using a .runsettings file
Hey, I think this issued I recently posted is related/same #1024
I think the main problem is the same. thanks for that
Any progress on this topic? We cannot upgrade from 2.2.3 to 2.2.8 until this is fixed :(
This is not a bug, mstest now reports data driven tests as single tests. This was done to bring parity to how xunit and nunit does it.
@Haplois There are few duplicates of this issue. Could you link them all to a single one, and post an update on when the work on AzDO will be finished, please?
Also correct my statement above if I am wrong :)
@nohwnd
The ONLY reason I and many other engineers use MSTest is because of the seamless integration with Azure DevOps Test Hub.
If you cannot maintain this integration you are just undermining the project.
Here are my concerns:
- The team broke this integration a year ago and it has yet to be addressed
- The interesting (wrong) assertion that this is "not a bug"
- The trx report format has fundamentally changed and that doesn't seem to justify a major or minor version bump
- I know this project and the Azure DevOps Visual Studio Test task are separate projects, but this kind of thing has the stink of the bad old days of Microsoft. - AKA Microsoft cannot seem to even work with itself
As far as I can tell here are the related/duplicate issues:
- https://github.com/microsoft/testfx/issues/860
- https://github.com/microsoft/testfx/issues/955
- https://github.com/microsoft/testfx/issues/992
- https://github.com/microsoft/testfx/issues/1023
- https://github.com/microsoft/testfx/issues/1026
- https://github.com/microsoft/testfx/issues/1035
Duly noted. Believe me, I am not happy about this either. We are looking into conditionally reverting this change until it is fixed in azdo.
@nohwnd - If I may make a suggestion: I would love to see splitting or consolidating tests being optional, much like parallelization. Maybe using an attribute such as "DataDrivenToSingleTest". This way users have the option of how they want to group such tests. *This would follow the same basic pattern as the "DoNotParallelize" attribute.
Example:
[DataTestMethod]
[DataDrivenToSingleTest] // Allow users to set this on the test, test class, and in the run settings file like you can for parallel
[DataRow("ConfigJsonEnvRunOverride", "OVERRIDE")]
[DataRow("OverrideOnly", "OVERRIDE")]
This is not a bug, mstest now reports data driven tests as single tests. This was done to bring parity to how xunit and nunit does it.
@Haplois There are few duplicates of this issue. Could you link them all to a single one, and post an update on when the work on AzDO will be finished, please?
Also correct my statement above if I am wrong :)
Hello @nohwnd Can you please provide link for some kind of documentation where this new behavior is explained? I couldnt find anything in MSTEST release notes. I would like something to grasp on when I explain this new behavior to my team.
Duly noted. Believe me, I am not happy about this either. We are looking into conditionally reverting this change until it is fixed in azdo.
Hoping to resolve this obstacle or to find an alternative solution for it. Our project has suffered because of that.
Hello @nohwnd Can you please provide link for some kind of documentation where this new behavior is explained? I couldnt find anything in MSTEST release notes. I would like something to grasp on when I explain this new behavior to my team.
I would also like to know, as I did not find the related changes by glancing over 2.2.4's release notes and commits.
Are there any updates to this as yet? also getting very frustrated and using work arounds is less than ideal
@Haplois @Evangelink did you get any update from AzDo about the progress of the fix? I know we talked with them a while ago.
They have a fix, and expect to ship it next sprint. I asked about exact date.
Getting the next sprint date would be great. As far as I can see, this change significantly breaks all test harnesses which are heavily Data Driven. I.e., test harness which have many rows in Excel, Access, or a Database. Teams following that methodology may have 100 test cases in one spreadsheet, with differ data rows of the spreadsheet testing different features, all going through one test method. While this is not ideal, a lot of teams/organizations perform testing. If one of these rows fails, the entire test fails? That does really skew the results.
Related well-reasoned posts on this issue include: https://developercommunity.visualstudio.com/t/vstest-test-publication-miscounts-test-cases-and-c/909375 https://developercommunity.visualstudio.com/t/tests-tab-on-build-results-page-shows-wrong-test-c/1385811
Looking forward to seeing what the fix is. If is necessary to bring MSTest.exe in line with NUnit and XUnit, there should be a command line flag to allow for backward compatibility with the old approach.
I spoke with the product team again, the fix in release pipeline and will be deployed within this sprint (which will be ending on 12th of September). I couldn't get an exact date.
/cc @dogirala from the product team.
Hi @charlesdwmorrison! My mistake, I should not have put any milestone on this ticket, as far as we know, sprint ended on 12th of September and roll out is supposed to have happened (@dogirala could you confirm?).
If you have tested and you are still experiencing the issue, please let us know and we will ping Azure DevOps team.
@Evangelink , @charlesdwmorrison I see the fix has been deployed but somehow it didn't reflect in microsoft which is strange. For microsoft org I still see old PTR task version which is 2.203.0, ideally it should be 2.210.0. I am checking with my team and concerned deployments team. Will let you know as soon as I find out something on this.
Other ex pipeline:
Got to know that this is expected as the deployments are still not completely done and they are in progress. Eta is by end of next week.
I will move forward and close this ticket. Please if anyone is still experiencing issue either comment below or create a new ticket and we will handle it.
I'm confused. I thought there would be a new version of MSTest released to fix this issue. I'm not seeing a new version on NuGet.org.
https://www.nuget.org/packages/MSTest.TestFramework/2.3.0-preview-20220810-02#versions-body-tab
I am still seeing the same test case count as I have been for the past two years. In the test shown in the screenshot below, there are 126 data driven tests (in the form of an Excel spreadsheet). But the test results are only showing 1 test. Where would the data driven counts show up?
I'm confused. I thought there would be a new version of MSTest released to fix this issue. I'm not seeing a new version on NuGet.org.
https://www.nuget.org/packages/MSTest.TestFramework/2.3.0-preview-20220810-02#versions-body-tab
I'm also a bit confused. My understanding now is that the discussed fix from above is not a fix to MSTest but the test results view in the build results, i.e. that the new test results format will now be visualized similarly to before the MSTest test results format change, right? How can I make use of this when using Azure DevOps Server (i.e. on-prem)?
For on-prem I think you'll need to download the latest agent and configure it for the new changes to be included
Hey there! Sorry the messages weren't super clear.
The issue raised in this ticket wasn't linked to a problem in MSTest per say but it was linked to some change of behavior in MSTest (it should not have been done in a patch version) that wasn't reflected on AzDo side. AzDo team updated the behavior on their side and we were waiting for it to be deployed (according to @dogirala it was done 2 days ago).
Regarding on-prem instances, @dogirala could you please let us know what's the process?
@charlesdwmorrison Could you confirm that you are using non on-prem AzDo?
I will make some more manual tests to confirm issue is correctly fixed and will post my results here. In the meantime, I will reopen the ticket.
I believe that is correct, Our AzDo instance is hosted by Microsoft, but we have our own agent machines. So that is "not on prem."
For on-prem instances, as per my knowledge you need to configure a new Agent or update the existing Agent in the AgentPool section of organization for the latest changes to pick up:
new agent:
Update agent:
