Selenium attachments are not saved to Azure DevOps report when running tests in parallel

[NetanelMosheCohen]

Description

I have a web automation project using Selenium, C#, MSTest V1, the report integrated with Azure DevOps. In the past I ran the tests sequentially and everything worked well - I saw the tests snapshots in the Azure DevOps integrated report. When I changed my project to run in parallel, I don't see my snapshots anymore rather get this error message (attached image):

TestContext Messages: Value cannot be null. Parameter name: path

Now it's occurs only when running in parallel.

Steps to reproduce

To run the tests in parallel, I added "parallelTestCount="8" to "Execution" tag in .TestSettings file. To take snapshots of a test I'm using the following code:

public TestContext currentTestContext { get; set; }

public void AddAttachments(DriverManager driverManager) { Screenshot screenshot = ((ITakesScreenshot)driverManager.Driver).GetScreenshot(); string attachmentsPath = Directory.GetCurrentDirectory() + currentTestContext.TestName + ".png"; screenshot.SaveAsFile(attachmentsPath); currentTestContext.AddResultFile(attachmentsPath); }

Every test class inherits from the TestBase class that contains the code above. In every test class, the function above defined as [TestCleanup] so the snaphots is taken in the end of the test. The tests are running through Azure DevOps CI on a Virtual Machine (Windows)

Expected behavior

The snapshots that taken in the end of the test should be displayed in the "Attachments" tab in Azure DevOps report.

Actual behavior

The snapshots are not displayed, the following error message appears:

TestContext Messages: Value cannot be null. Parameter name: path

I saw that another people faced this issue years ago but they didn't get a real solution: https://social.msdn.microsoft.com/Forums/en-US/29b74cdd-41e8-45ce-a599-08b387cec8c5/paralleltestcount-and-addresultfile-problem-?forum=vsunittest

https://stackoverflow.com/questions/38450326/c-sharp-selenium-mstest-unable-to-add-a-result-file-using-testcontext-ad

https://social.msdn.microsoft.com/Forums/en-US/7d126d85-daeb-41e7-a75c-5c35646d6dd4/results-in-folder-is-deleted-if-tests-are-run-in-parallel-and-addresultfile-method-does-not-work?forum=vsunittest

Environment

Win 10, vstest.console version 16.1, MSTest version both 1.4 and 2 Selenium, C#, .NET Framework

[elgatov]

@NetanelMosheCohen try with:

public void AddAttachments(DriverManager driverManager)
{
    Screenshot screenshot = ((ITakesScreenshot)driverManager.Driver).GetScreenshot();
    string attachmentsPath = Directory.GetCurrentDirectory() + "\\" + currentTestContext.TestName +
    ".png";
    screenshot.SaveAsFile(attachmentsPath);
    currentTestContext.AddResultFile(attachmentsPath);
}

[NetanelMosheCohen]

@elgatov Thanks but it didn't help, I don't think that it's because of the path, I tried a couple of paths... I think that the issue is more complicated.

[karanjitsingh]

@NetanelMosheCohen, sorry for the delayed response, could you post full logs for this along with the generated trx file?

[NetanelMosheCohen]

@karanjitsingh Do you mean to the logs that generated in Azure Devops when running the tests (VsTest logs only)?

[NetanelMosheCohen]

@karanjitsingh if so, attached.

SeleniumLogs.zip

[karanjitsingh]

@NetanelMosheCohen Two things here-

• .Testsettings does not work with MSTestV2. Try running parallelism with a runsettings file. https://github.com/Microsoft/testfx-docs/blob/master/RFCs/004-In-Assembly-Parallel-Execution.md
• Is it possible the test throws before your AddAttachments code gets hit? Try logging something to console along with the name of the running test in that block, see if it shows up in the stdout.

[NetanelMosheCohen]

@karanjitsingh

1. I tried to upgrade MsTest to V2 with RunSettings, but [DataSource] is not available in MsTest V2 so I gave up, because I don't want to update each test with new function of reading test data. I working with MsTest V1 and TestSettings file and it is working well. Also note that adding snapshots with non-parallel execution is works so the issue appears only when running in parallel.

2. I checked it now, I wrote to the log in each step in this function, but the error is shown after the last line: currentTestContext.AddResultFile(attachmentsPath);

[karanjitsingh]

@NetanelMosheCohen

TestContext Messages:
Value cannot be null.
Parameter name: path

Can you check who's printing this? Maybe your tests are catching an exception but not printing out the stack.

[NetanelMosheCohen]

@karanjitsingh AddResultFile() is printing this error as you can see in your docs: https://github.com/microsoft/testfx/blob/master/src/Adapter/PlatformServices.NetCore/Services/NetCoreTestContextImplementation.cs

In the bug description there are 3 links of people that get the exact same issue so the problem is not in my specific tests.

[ShreyasRmsft]

@NetanelMosheCohen this is AddResultFile implementation in MsTestV2. The code might not be the same for MsTestV1

Does this repro with MsTestV2 also? I'm not sure if we will be making any fixes in MsTestV1. If there is a bug with MsTestV2 we will addressing it in V2.

Could you mock the data and try for MsTestV2?

[NetanelMosheCohen]

@ShreyasRmsft The code is identical in MsTestV1 as I saw in old tickets. Also, Microsoft themselves gave me the code as you can see here in the comments: https://azuredevopslabs.com/labs/vstsextend/selenium/

As I wrote above, I tried to upgrade MsTest to V2 with RunSettings, but [DataSource] is not available in MsTest V2 so I gave up, because I don't want to update each test with new function of reading test data, I have hundreds of tests. I working with MsTest V1 and TestSettings file and it is working well. Also note that adding snapshots with non-parallel execution is works so the issue appears only when running in parallel.

[ShreyasRmsft]

@NetanelMosheCohen I reiterate: the link provided is the code for mstestV2. Mstestv1 code is not in any public repository.

secondly we will not be making any fixes in Mstest v1. MsTest v2 was created to address the short comings in v1.

As for [DataSource] can you tell me where you are getting the data from?

[NetanelMosheCohen]

@ShreyasRmsft I'm getting the data from xml files

[ShreyasRmsft]

@NetanelMosheCohen there are ways to get the data from xml files using MsTestV2s custom data source extensibility model.

But yeah if you can verify if the issue with AddResultFile is resolved for you with MsTestV2 (maybe create a dummy set of tests and run them in parallel) then i can go ahead and help with the data driving using xml files with mstest v2.

If it turns out that it's broken with mstest v2 (please provide a sample repro project) then we'll try and fix it in mstestv2

[NetanelMosheCohen]

@ShreyasRmsft Please can you provide me a sample code how to read the test data from xml using custom data source extensibility model?

[ShreyasRmsft]

This is something we ourselves authored for reading test case data from .txt files The same logic can be applied for xml files. You'd need to make a few changes but the idea is that using this you can do anything in the method and just yield a list of objects that are used to data drive your tests.

https://github.com/nigurr/TestResultParser/blob/master/Agent.Plugins.UnitTests/TestResultParserTestBase.cs#L163

Here's a test class that uses the above base class method to read test data from the .txt files

https://github.com/nigurr/TestResultParser/blob/master/Agent.Plugins.UnitTests/MochaTestResultParserTests/MochaTestResultParserTests.cs

Clone the project locally and take a look at the test project, it's a great example for leveraging mstest v2's extensibility.

[NetanelMosheCohen]

Thanks!