SteveGilham
altcover runner --OutputFile gives incorrect summary output
Repo Steps
First I instrumented the binaries using altcover:
PS D:\TestProject.Console> altcover `
--inputDirectory=D:\TestProject.Console\bin\Release\net8.0 `
--outputDirectory=D:\TestProject.Console\bin\Instrumented\net8.0
Creating folder D:\TestProject.Console\bin\Instrumented\net8.0\
Instrumenting files from D:\TestProject.Console\bin\Release\net8.0\
Writing files to D:\TestProject.Console\bin\Instrumented\net8.0\
=> D:\TestProject.Console\bin\Release\net8.0\AltCover.Monitor.dll
Coverage Report: D:\TestProject.Console\coverage.xml
D:\TestProject.Console\bin\Instrumented\net8.0\AltCover.Monitor.dll
<= AltCover.Monitor, Version=8.8.0.0, Culture=neutral, PublicKeyToken=null
Next I run the app using altcover runner with --outputFile.
PS D:\TestProject.Console>cd bin\Instrumented\net8.0
PS D:\TestProject.Console\bin\Instrumented\net8.0> altcover runner `
--recorderDirectory=D:\TestProject.Console\bin\Instrumented\net8.0 `
--outputFile=D:\TestProject.Console\bin\Instrumented\net8.0\coverage.xyz.xml.acv `
--executable=dotnet -- TestProject.Console.dll
Beginning run...
Command line : '"dotnet" TestProject.Console.dll'
[command output]
Getting results...
... D:\TestProject.Console\coverage.xml.0.acv (50,446b)
5,083 visits recorded in 00:00:00.2770883 (18,344 visits/sec)
A total of 5,083 visits recorded
Coverage statistics flushing took 0.33 seconds
Visited Classes 0 of 346 (0)
Visited Methods 0 of 2419 (0)
Visited Points 0 of 14002 (0)
Visited Branches 0 of 8422 (0)
Maximum CRAP score 0
==== Alternative Results (includes all methods including those without corresponding source) ====
Alternative Visited Classes 0 of 359 (0)
Alternative Visited Methods 0 of 2535 (0)
Alternative maximum CRAP score 0
Expected Results
- Output should be saved to
D:\TestProject.Console\bin\Instrumented\net8.0\coverage.xyz.xml.acv
Actual Results
- Output is saved as
D:\TestProject.Console\coverage.xml.0.acv - Notice
--ouputFileisn't even used and it is saved in the directory wherealtcoverinstrumentation command is run.
I tried looking through the code source code, and only found it using --outputFile with regard to JSON here. However, I admit I don't know the project well enough and I'm not familiar enough with F#.
I just realized that the Getting results... line might not be the place where coverage data is be saved to.
I did find that the coverage file at the --outputFile location, contains only: <null />. I'm not quite sure what I am doing wrong.
The root of the problem is naming the output file with .acv suffix, in particular a name of the form [Coverage Report].*.acv which used by altcover for intermediate files; your chosen file name is getting caught up in this section of code.
The simple fix is to keep XML output files as .xml and JSON files as .json.
Ignore the above, I misread this as coverage.xml.xyz.acv, which would be deleted.
With a simple example,
using System;
// test program
// for report generation
namespace TouchTest
{
internal class Program
{
private static void Main(string[] args)
{
var now = DateTime.Now;
if (now.Year > 2000)
{
Console.WriteLine("Where is my rocket pack? " + String.Join("*", args));
}
else
{
Console.WriteLine("Twentieth Century, boy!");
}
}
}
}
following your recipe
I get
D:\Github\TestProject1\TestProject.Console\bin\Instrumented\net8.0\coverage.xyz.xml.acv looking like this
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<CoverageSession xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Summary numSequencePoints="5" visitedSequencePoints="3" numBranchPoints="3" visitedBranchPoints="2" sequenceCoverage="60" branchCoverage="66.67" maxCyclomaticComplexity="2" minCyclomaticComplexity="1" visitedClasses="1" numClasses="1" visitedMethods="1" numMethods="1" minCrapScore="2.26" maxCrapScore="2.26" />
<Modules>
<Module hash="26-6D-BA-6A-B8-F6-50-C7-FD-75-42-A9-3A-9E-E5-DD-E6-DD-66-84">
<Summary numSequencePoints="5" visitedSequencePoints="3" numBranchPoints="3" visitedBranchPoints="2" sequenceCoverage="60" branchCoverage="66.67" maxCyclomaticComplexity="2" minCyclomaticComplexity="1" visitedClasses="1" numClasses="1" visitedMethods="1" numMethods="1" minCrapScore="2.26" maxCrapScore="2.26" />
<ModulePath>D:\Github\TestProject1\TestProject.Console\bin\Release\net8.0\TestProject.Console.dll</ModulePath>
<ModuleTime>2024-04-29T06:23:23.005386Z</ModuleTime>
<ModuleName>TestProject.Console</ModuleName>
<Files>
<File uid="1" fullPath="D:\Github\TestProject1\TestProject.Console\Program.cs" />
</Files>
<Classes>
<Class>
<Summary numSequencePoints="5" visitedSequencePoints="3" numBranchPoints="3" visitedBranchPoints="2" sequenceCoverage="60" branchCoverage="66.67" maxCyclomaticComplexity="2" minCyclomaticComplexity="1" visitedClasses="1" numClasses="1" visitedMethods="1" numMethods="1" minCrapScore="2.26" maxCrapScore="2.26" />
<FullName>TouchTest.Program</FullName>
<Methods>
<Method visited="true" cyclomaticComplexity="2" nPathComplexity="2" sequenceCoverage="60" branchCoverage="66.67" isConstructor="false" isStatic="true" isGetter="false" isSetter="false" crapScore="2.26">
<Summary numSequencePoints="5" visitedSequencePoints="3" numBranchPoints="3" visitedBranchPoints="2" sequenceCoverage="60" branchCoverage="66.67" maxCyclomaticComplexity="2" minCyclomaticComplexity="2" visitedClasses="0" numClasses="0" visitedMethods="1" numMethods="1" minCrapScore="2.26" maxCrapScore="2.26" />
<MetadataToken>100663297</MetadataToken>
<Name>System.Void TouchTest.Program::Main(System.String[])</Name>
<FileRef uid="1" />
<SequencePoints>
<SequencePoint vc="1" uspid="4" ordinal="0" offset="0" sl="12" sc="4" el="12" ec="27" bec="0" bev="0" fileid="1" />
<SequencePoint vc="1" uspid="3" ordinal="1" offset="6" sl="13" sc="4" el="13" ec="24" bec="2" bev="1" fileid="1" />
<SequencePoint vc="1" uspid="2" ordinal="2" offset="20" sl="15" sc="5" el="15" ec="77" bec="0" bev="0" fileid="1" />
<SequencePoint vc="0" uspid="1" ordinal="3" offset="47" sl="19" sc="5" el="19" ec="50" bec="0" bev="0" fileid="1" />
<SequencePoint vc="0" uspid="0" ordinal="4" offset="57" sl="21" sc="3" el="21" ec="4" bec="0" bev="0" fileid="1" />
</SequencePoints>
<BranchPoints>
<BranchPoint vc="1" uspid="0" ordinal="0" offset="18" sl="13" path="0" offsetend="20" fileid="1" />
<BranchPoint vc="0" uspid="1" ordinal="1" offset="18" sl="13" path="1" offsetend="47" fileid="1" />
</BranchPoints>
<MethodPoint xsi:type="SequencePoint" vc="1" uspid="4" ordinal="0" offset="0" sl="12" sc="4" el="12" ec="27" bec="0" bev="0" fileid="1" />
</Method>
<Method visited="false" cyclomaticComplexity="1" nPathComplexity="0" sequenceCoverage="0" branchCoverage="0" isConstructor="true" isStatic="false" isGetter="false" isSetter="false" crapScore="2">
<Summary numSequencePoints="0" visitedSequencePoints="0" numBranchPoints="0" visitedBranchPoints="0" sequenceCoverage="0" branchCoverage="0" maxCyclomaticComplexity="1" minCyclomaticComplexity="1" visitedClasses="0" numClasses="0" visitedMethods="0" numMethods="0" minCrapScore="2" maxCrapScore="2" />
<MetadataToken>100663298</MetadataToken>
<Name>System.Void TouchTest.Program::.ctor()</Name>
<SequencePoints />
<BranchPoints />
<MethodPoint vc="0" uspid="100663298" ordinal="0" offset="0" />
</Method>
</Methods>
</Class>
</Classes>
</Module>
</Modules>
</CoverageSession>
The bug comes in the summary reporting, which still seems to be reading from the untouched coverage file -
Coverage statistics flushing took 0.03 seconds
Visited Classes 0 of 1 (0)
Visited Methods 0 of 1 (0)
Visited Points 0 of 5 (0)
Visited Branches 0 of 3 (0)
Maximum CRAP score 0
==== Alternative Results (includes all methods including those without corresponding source) ====
Alternative Visited Classes 0 of 1 (0)
Alternative Visited Methods 0 of 2 (0)
Alternative maximum CRAP score 0
The reference
Getting results...
... D:\Github\TestProject1\TestProject.Console\coverage.xml.0.acv (99b)
is to the raw binary format coverage data output during the run.
The root of the problem is naming the output file with .acv suffix, in particular a name of the form [Coverage Report].*.acv which used by altcover for intermediate files; your chosen file name is getting caught up in this section of code.
I think this ended up being my problem. Although specifying --outputFile did not increase my coverage as I had expected. I thought maybe between runs of altcover runner on different executables, it was overwriting the previous report but perhaps maybe that wasn't the case.
It might be a good idea to:
- Have a warning if
--outputFileends in.acv. - Print the path to the
--outputFilein the logs/summary if the user specified it.
The use case for the separate output file is where you want to compare the coverage of two or more different (typically operational test) runs so you instrument once, to create a zero-coverage output file O, run once with a new output file O1, and a second time with output file O2 and can compare them.
If you keep using the same output file, then it will get overwritten with O plus whatever the latest test run did, in much the same way that if you don't give an output file, the original O gets updated by adding the latest test run each time.
I think my coverage problem was/is unrelated to what I initially thought it was. It ended up being that each instrumented directory (I have a solution with multiple console apps), needed to have its own --report= with altcover otherwise only the last report was uploaded. For whatever reason --outputFile= with altcover runner didn't make much of a difference at this level or I didn't get it working right.
I'm not sure if my ideas in my last comment were of any value, but feel free to close this issue. Thanks again for all your help!