Avoid guid at the end of outputDirectory

[MarcoRossignoli]

On Coverlet we have some users complain about the dynamic folder name generated for reports file. I did some check on code and seem that guid is "mandatory" and not removable.

https://github.com/microsoft/vstest/blob/master/src/Microsoft.TestPlatform.Common/DataCollection/DataCollectionAttachmentManager.cs#L101-L125

Could be useful add a parameter, maybe in runsetting like <DeterministicOutputdirectory>true/false</DeterministicOutputdirectory> and in that case avoid to concat guid.ToString().

Dynamic folder makes CI more complex.

Issues refs https://github.com/tonerdo/coverlet/issues/500 https://github.com/tonerdo/coverlet/issues/767

cc: @DaleyKD

[nohwnd]

I don't know the reasons for the guid but I would rather avoid yet another option. I see the coverage file described in the TRX output, but even thought there is a href to it, it does not contain the guid. I also don't see the guid anywhere else in the repo. Imho the best course of action is to provide a way that will allow the test result to be programatically processed.

When you run with CC in azure pipeline, do you see the coverage file automatically uploaded? maybe they have this already figured out.

[nohwnd]

Admittedly the trx, and xml in general is not the easiest to process, but it's a start, and it is generic format that we don't have to re-invent.

[MarcoRossignoli]

When you run with CC in azure pipeline, do you see the coverage file automatically uploaded? maybe they have this already figured out.

Looking at publish result task https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml seem that a glob filter is used

- task: PublishTestResults@2
  inputs:
    testRunner: VSTest
    testResultsFiles: '**/*.trx'
    failTaskOnFailedTests: true

[MarcoRossignoli]

Admittedly the trx, and xml in general is not the easiest to process, but it's a start, and it is generic format that we don't have to re-invent.

I'm ok with file format, but non deterministic path add some work to do in some scenario, tipical "find right file and move to".

[nohwnd]

I meant it more in the direction of returning a simple object that can be programatically processed to get to the info, instead of forcing the naming conventions to be deterministic. For example in Pester I simply return a result object that the caller can process. Here we could do something similar. Say you'd want to upload your CC file:

$result = dotnet test --collect="Code Coverage" ....params.... --json-result 
($result | ConvertFrom-Json).Attachements | where Type -eq "Code Coverage" | Invoke-RestMethod -Method PUSH

This would be the end product where we would have to come up with the result etc... So a more achievable goal would be to hookup the trx file and read the path to the CC file from there if it was correct.

[MarcoRossignoli]

$result = dotnet test --collect="Code Coverage" ....params.... --json-result

If I don't use powershell what does --json-result produce?Is it a json on standard output?

[nohwnd]

Yes, it was just an exam

---

Od: Marco Rossignoli [email protected] Odesláno: Wednesday, April 1, 2020 8:05:48 PM Komu: microsoft/vstest [email protected] Kopie: Jakub Jareš [email protected]; Comment [email protected] Předmět: Re: [microsoft/vstest] Avoid guid at the end of outputDirectory (#2378)

$result = dotnet test --collect="Code Coverage" ....params.... --json-result

Are you forcing powershell usage?If I don't use powershell what does --json-result produce?Is it a json on standard output?

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/microsoft/vstest/issues/2378#issuecomment-607406005, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABLYLYIFL7VICTTBJS7R7ILRKN67ZANCNFSM4LVPYFMA.

[nohwnd]

Yes json, it was just an example to show a concept. I am not forcing powershell on anyone.

---

Od: Marco Rossignoli [email protected] Odesláno: Wednesday, April 1, 2020 8:05:48 PM Komu: microsoft/vstest [email protected] Kopie: Jakub Jareš [email protected]; Comment [email protected] Předmět: Re: [microsoft/vstest] Avoid guid at the end of outputDirectory (#2378)

$result = dotnet test --collect="Code Coverage" ....params.... --json-result

Are you forcing powershell usage?If I don't use powershell what does --json-result produce?Is it a json on standard output?

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/microsoft/vstest/issues/2378#issuecomment-607406005, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABLYLYIFL7VICTTBJS7R7ILRKN67ZANCNFSM4LVPYFMA.

[MarcoRossignoli]

What if we have more than one collector https://github.com/Microsoft/vstest-docs/blob/master/docs/analyze.md#enabledisable-a-datacollector? A json array with different Type(collector friendly name)?

[ViktorHofer]

Just stumbled upon this.

I don't know the reasons for the guid but I would rather avoid yet another option.

I disagree. VSTest should respect the specified ResultsDirectory and CoverageFileName for data collectors and not create intermediate directories between. I can do that for loggers already so I would argue DataCollectors to behave the same:

<RunSettings>
  <RunConfiguration>
     <!-- Directory for test run reports. E.g. trx, coverage etc. -->
    <ResultsDirectory>.\</ResultsDirectory>
  </RunConfiguration>
  <LoggerRunSettings>
    <Loggers>
      <Logger uri="logger://Microsoft/TestPlatform/TrxLogger/v1">
        <Configuration>
          <LogFileName>testResults.trx</LogFileName>
        </Configuration>
      </Logger>
      <Logger uri="logger://Microsoft/TestPlatform/HtmlLogger/v1">
        <Configuration>
          <LogFileName>testResults.html</LogFileName>
        </Configuration>
      </Logger>
    </Loggers>
  </LoggerRunSettings>

I would expect the following behavior:

• LogFileName inside a DataCollector configuration node is respected:

<DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <CoverageFileName>coverage.xml</CoverageFileName>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>

• The specified ResultsDirectory is respected without any intermediate folders in it.

  <RunConfiguration>
     <!-- Directory for test run reports. E.g. trx, coverage etc. -->
    <ResultsDirectory>.\</ResultsDirectory>
  </RunConfiguration>

In this example you can even see that coverage is explicitly listed in the comment (which I took from the docs). Specifying the log file name seems to have worked in the past with the pre-coverlet code coverage collector: https://github.com/microsoft/vstest/issues/1957.

[ViktorHofer]

I just observed another misbehavior:

The coverage report is created twice:

• vihofer_DESKTOP-803H9R0_2020-04-22_06_59_42\In\DESKTOP-803H9R0\coverage.opencover.xml
• 4a408eed-656b-435d-a540-30c285d1c57a\coverage.opencover.xml

[MarcoRossignoli]

The coverage report is created twice:

@ViktorHofer I think it's a dup of https://github.com/microsoft/vstest/issues/2334

[ViktorHofer]

You are right :)

[ViktorHofer]

FWIW I currently need to clean the TestResults directory manually before VSTest is invoked to make sure that ReportGenerator picks the right coverage results file later. Ideally we would not need to do this.

[paulomorgado]

The directory where the coverage results will be published is required to exist. Otherwise it throws:

Data collector 'Code Coverage' message: Data collector caught an exception of type 'Microsoft.VisualStudio.Coverage.VanguardException': 'Running event not received from CodeCoverage.exe. Check eventlogs for failure reason.'. More details: ..

That's also bad.

[ps2goat]

I think I see the purpose for the dynamic folder that is appended-- if you run it at the root level against a solution with multiple test projects, each project is in its own dynamic GUID folder.

A better solution would be to at least use the project name in this scenario. These names are predictable and should be unique: TestResults/ProjectA/<coveragefile>.xml TestResults/Projectb/<coveragefile>.xml

...unless there's some weird nesting in folders that would allow duplicate project names. That sounds like an edge case that the user would have to fix, though, and in that case you could use GUID folders per duplicated project or something.

[ArchieCoder]

Spent the evening doing tests here and there trying to understand why I was getting random folders. Glad that I finally found this, but sad this is not implemented. This is definitively not CI/CL friendly.

[hauntingEcho]

This is also not IDE friendly, as there is no deterministic way to find the most recent result folder. Coverage highlighting tools (VSCode Coverage Gutters, in my case) can't be effectively configured to work with this behavior.

[tonyhallett]

dotnet test standard output provides the generated folder name

"Starting test execution, please wait...\r\nLogging Vstest Diagnostics in file: C:\Users\tonyh\Source\Repos\DataCollectorXUnit\XUnitTestProject1\bin\Debug\netcoreapp3.1\fine-code-coverage\coverage-tool-output\diagnostics.log\r\nA total of 1 test files matched the specified pattern.\r\n\r\n Attachments:\r\n C:\Users\tonyh\Source\Repos\DataCollectorXUnit\XUnitTestProject1\bin\Debug\netcoreapp3.1\fine-code-coverage\coverage-tool-output\**7ba6447d-a89f-4836-bffc-aeb4799e48ab\coverage.cobertura.xml**r\nPassed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 7 ms - XUnitTestProject1.dll (netcoreapp3.1)"

The diagnostic log also has the generated folder name

TpTrace Verbose: 0 : 4712, 10, 2021/02/21, 14:58:38.317, 1073211811845, vstest.console.dll, DataCollectionRequestSender.SendAfterTestRunStartAndGetResult : Received message: (DataCollection.AfterTestRunEndResult) -> [ { "Uri": "datacollector://Microsoft/CoverletCodeCoverage/1.0", "DisplayName": "XPlat code coverage", "Attachments": [ { "Description": "", "Uri": "file://C:/Users/tonyh/Source/Repos/DataCollectorXUnit/XUnitTestProject1/bin/Debug/netcoreapp3.1/fine-code-coverage/coverage-tool-output/7ba6447d-a89f-4836-bffc-aeb4799e48ab/coverage.cobertura.xml" } ] } ]

Agree that should avoid the guid though.

[psvaiter]

Is there any solution found for this issue? I'm suffering from the same problem, while msbuild integration allows a specification of ouput dir. Why not vstest?

[ArchieCoder]

@psvaiter Here is my solution. I call this executable before using the coverage file. The tool search for the coverage file and copy at the specified file path.

`using System; using System.IO;

namespace CopyCoverageFile { class Program { static void Main(string[] args) { if (args.Length == 2) { var files = Directory.GetFiles(args[0], "*.coverage", SearchOption.AllDirectories);

            var path = Path.Combine(args[1], "YOUR_TEST.coverage");

            Directory.CreateDirectory(args[1]);

            File.Copy(files[0], path);

            Console.WriteLine("Is coverage created? " + File.Exists(path));
        }
        else if (args.Length == 1)
        {
            Console.WriteLine($"Is new {args[0]} exist? " + File.Exists(args[0]));
        }
    }
}

}`

[Cobster]

Hoping this may help a few people out there, I use the following snippet of PowerShell in my development scripts to extract the unit test results paths and feed them to report generator.

$TestOutput = dotnet test --collect "XPlat Code Coverage"
$TestReports = $TestOutput | Select-String coverage.cobertura.xml | ForEach-Object { $_.Line.Trim() } | Join-String -Separator ';'
dotnet reportgenerator "-reports:$TestReports" "-targetdir:./CoverageReport" "-reporttype:Html"

[psvaiter]

@ArchieCoder I tried to use MSBuild integration and that worked as expected. Besides that I could use the Fine Code Coverage extension for Visual Studio that seems to depend on coverlet.msbuild package.

Here's the command I run:

dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=TestResults\

This way I stopped using the vstest.

[keegandent]

This is also not IDE friendly, as there is no deterministic way to find the most recent result folder. Coverage highlighting tools (VSCode Coverage Gutters, in my case) can't be effectively configured to work with this behavior.

I am right there with you. I'm not sure what genius thought an artifact directory's name should be non-deterministic, but it's completely ludicrous to me. Nobody has, from my reading, presented a single good argument as to why you'd want a GUID for this. I could see the argument for a timestamp, but honestly, if somebody wants their artifacts in a GUID-named folder, make them do it themselves as its a lot easier to go deterministic->random than vice-versa. I'm not sure why those who want consistency are the ones being punished.

[jasonterando]

FWIW, I'm using dotnet and Visual Studio for .NET Core applications. I wanted to execute my unit tests and generate a coverage report, with log files going into .results/log and the coverage report in .results/report. I ended up going the "shell script" route (hey, we're all *nix users with WSL now, right?). Here's what works for me:

Step #1: Install Report Generator as a local tool:

dotnet new tool-manifest
dotnet tool install dotnet-reportgenerator-globaltool

Step #2: Make sure your project has coverlet collector:

dotnet add package coverlet.collector

Step #3: Create a Bash script file with below contents, tweak to your liking. Don't forget to chmod +x it. The script creates the .results folder if it does not exist, cleans up any previous files in .results, and then executes unit tests and report generation. It exits with the exit code from the unit tests, so if you are running this in a CI/CD pipeline, you can continue (or not) if the unit tests pass.

#!/bin/sh
mkdir -p .results
rm -r -f .results/*
dotnet test --collect:"XPlat Code Coverage" -r .results/log -l trx
result=$?
dotnet reportgenerator -reports:`find .results -name coverage.cobertura.xml` -targetdir:.results/reports
if [ $result = 0 ]; then
    echo "All tests passed!"
else
    echo "Some tests failed :("
fi
exit $result

You could get beef this up if you wanted to only exit successfully (exit code of not zero) based upon code coverage, successful report generation, etc. but I have messed with this enough this morning ;)

[gix]

A better solution would be to at least use the project name in this scenario.

This is not enough since you can have a test project with multiple target frameworks.

[f00-beard]

I ran into same issue. This may help in some cases: You can deterministically generate a trx log file which contains a partial path and name of the .coverage file using: dotnet test --collect:"Code Coverage" --logger "trx;LogFileName=TestResults.trx" and you can parse the XML to get the file path using:

static string? GetAttachmentPath(string trxPath)
{
    var doc = new XmlDocument();
    doc.Load(trxPath);

    var namespaceManager = new XmlNamespaceManager(doc.NameTable);
    namespaceManager.AddNamespace("vs", "http://microsoft.com/schemas/VisualStudio/TeamTest/2010");

    XmlNode? firstCoverageAttachment = doc.DocumentElement?.SelectSingleNode(
        xpath: @"/vs:TestRun/vs:ResultSummary/vs:CollectorDataEntries/vs:Collector[@uri=""datacollector://microsoft/CodeCoverage/2.0""]/vs:UriAttachments/vs:UriAttachment[1]/vs:A",
        nsmgr: namespaceManager);

    return firstCoverageAttachment?.Attributes?["href"]?.Value;
}

[MarcoRossignoli]

Related issue https://github.com/microsoft/vstest/issues/2334#issuecomment-1156634841

[ViktorHofer]

@MarcoRossignoli @nohwnd is someone actively working on this? It's still causing unreasonable pain and the issue is upvoted quite a lot. Also, the milestone seems to be incorrect as 17.1 already shipped.

[Evangelink]

@ViktorHofer We are not working on this. I will bring it up at tomorrow's standup.