Code coverage not listing all missed commands

[mrboring]

1. General summary of the issue

When running code coverage in a file not all Missed commands are listed. Test code:

Test-Function.ps1

function Test-Function {
    try {
        Out-Null
    }
    catch {
        Write-Host 'Dummy'
        throw
    }

    foreach ($i in 0..3) {
        if ($true) {
            continue
        }
    }
}

Test-Function.Tests.ps1

. '.\Test-Function.ps1'

Describe -Name 'Test-Function' -Fixture {

    It -Name 'dummy' -Test {
        Mock -CommandName 'Write-Host' #-MockWith {Start-Sleep -Seconds 5}

        Test-Function
    }

}

Test.ps1

Set-StrictMode -Version Latest

Clear-Host

$testPath = '.\Test-Function.Tests.ps1'
$filePath = '.\Test-Function.ps1'

Invoke-Pester -Script $testPath -CodeCoverage $filePath

Output

Pester v4.10.1
Executing all tests in '.\Test-Function.Tests.ps1'

Executing script .\Test-Function.Tests.ps1

  Describing Test-Function
    [+] dummy 6ms
Tests completed in 92ms
Tests Passed: 1, Failed: 0, Skipped: 0, Pending: 0, Inconclusive: 0 

Code coverage report:
Covered 75.00% of 4 analyzed Commands in 1 File.
Missed command:

File              Class Function      Line Command
----              ----- --------      ---- -------
Test-Function.ps1       Test-Function    6 Write-Host 'Dummy'

2. Describe Your Environment

Pester version     : 4.10.1 C:\Users\<REMOVED>\Documents\PowerShell\Modules\Pester\4.10.1\Pester.psd1
PowerShell version : 7.0.0
OS version         : Microsoft Windows NT 10.0.18363.0

3. Expected Behavior

I expected Missd commands to include:

6        Write-Host 'Dummy'
7        throw
12       continue

4. Current Behavior

Missed commands only includes:

6        Write-Host 'Dummy'

[dlwyatt]

https://pester.dev/docs/usage/code-coverage/ :

A quick note on the "analyzed commands" numbers. You may have noticed that even though CoverageTest.ps1 is 17 lines long, Pester reports that only 5 commands are being analyzed for coverage. This is a limitation of the current implementation of the coverage analysis, which uses PSBreakpoints to track which commands are executed. Breakpoints can only be triggered by commands in PowerShell, which includes both calls to functions, Cmdlets and programs, as well as expressions and variable assignments. Breakpoints are not triggered by keywords such as else, try, or finally, or on opening or closing braces

[mrboring]

@dlwyatt So, in my test code it is possible to set PSBreakpionts on lines 7 and 12. I've done so in testing. Are the throw and continue keywords not covered?

[dlwyatt]

Correct, though you can often set a breakpoint on a line with throw if you pass it an argument. It's the expression that triggers the breakpoint though, not the throw keyword itself.

[mrboring]

@dlwyatt Thanks for the explanation, very helpful!

[nohwnd]

@dlwyatt was there maybe some change? Frankly I don't remember how exactly the BPs are setup for code coverage, so I might be trying it incorrectly here:

try {
    throw
}
catch { }

$bp.HitCount

This stops on the BP and reports 1.

[mrboring]

@dlwyatt, @nohwnd I've had a quick look at the source and wonder if the following change would be valid.

Function: Coverage.ps1\Get-CommandsInFile

Line: 195 before

        $predicate = {
            $args[0] -is [System.Management.Automation.Language.DynamicKeywordStatementAst] -or
            $args[0] -is [System.Management.Automation.Language.CommandBaseAst]
        }

Line: 195 after

        $predicate = {
            $args[0] -is [System.Management.Automation.Language.DynamicKeywordStatementAst] -or
            $args[0] -is [System.Management.Automation.Language.CommandBaseAst] -or
            $args[0] -is [System.Management.Automation.Language.BreakStatementAst]  -or
            $args[0] -is [System.Management.Automation.Language.ContinueStatementAst]  -or
            $args[0] -is [System.Management.Automation.Language.ExitStatementAst]  -or
            $args[0] -is [System.Management.Automation.Language.ReturnStatementAst]  -or
            $args[0] -is [System.Management.Automation.Language.ThrowStatementAst]
        }

I've done some simple tests and it works. I've included statements that can exist by themselves. I may have missed some. I don't know if there are implications to making this change.

Updated Test-Function.ps1 file

function Test-Function {
    try {
        Out-Null

        if ($false) {
            break
        }

        if ($false) {
            exit
        }

        if ($false) {
            return
        }
    }
    catch {
        Write-Host 'Dummy'
        throw
    }

    foreach ($i in 0..3) {
        if ($false) {
            continue
        }
    }
}

Results of code coverage:

Pester v4.10.1
Executing all tests in '.\Test-Function.Tests.ps1'

Executing script .\Test-Function.Tests.ps1

  Describing Test-Function
    [+] dummy 26ms
Tests completed in 108ms
Tests Passed: 1, Failed: 0, Skipped: 0, Pending: 0, Inconclusive: 0 

Code coverage report:
Covered 50.00% of 12 analyzed Commands in 1 File.
Missed commands:

File              Class Function      Line Command
----              ----- --------      ---- -------
Test-Function.ps1       Test-Function    6 break
Test-Function.ps1       Test-Function   10 exit
Test-Function.ps1       Test-Function   14 return
Test-Function.ps1       Test-Function   18 Write-Host 'Dummy'
Test-Function.ps1       Test-Function   19 throw
Test-Function.ps1       Test-Function   24 continue

[nohwnd]

Looks good. Please wait for a bit for confirmation from Dave if there will be any :)

Then please PR it with the example that you have as a test. You might need to fix the other tests, and maybe there are going to be some inconsistencies across PowerShell versions. I am prepared to help out with that in the PR.

If you are not interested in PRing it, let me know I will mark it as up for grabs. 🙂

[mrboring]

@nohwnd I'm in the process of testing the code. I've found an issue with the return keyword. It is not being detected as hit. So I've been looking through the source code. I've found where breakpoints are created, and can see that a breakpoint is set in the return line.

There is also $Pester.CommandCoverage. Each item has a BreakPoint property. This property includes a property of HitCount. The code that appears to update this is:

Pester.psm1\Invoke-Pester\Line 1111:

& $Path @Parameters @Arguments

This just executes the tests. It is inside a scriptblock that is called at line: 1137:

$testOutput = & $invokeTestScript -Path $testScript.Path -Script $testScript.Script -Arguments $testScript.Arguments -Parameters $testScript.Parameters -Set_ScriptBlockHint $script:SafeCommands['Set-ScriptBlockHint']

After this line is executed the HitCount is updated for commands that are hit. I can't figure out how it is doing this. Can you point me in the right direction.

[nohwnd]

This is how it works I think:

$sb = {
    try {
        throw
    }
    catch { }

    return
}

# in Enter-CoverageAnalysis -> New-CoverageBreakpoint
$params = @{
    Script = $sb.File
    Line   = 3
    Column = 9
    Action = { }
}

$breakpoint = @(
    (Set-PSBreakPoint -Script $sb.File -Line 3 -Column 9 -Action { }),
    # not hit when we specify column, but hit when we don't specify column
    (Set-PSBreakPoint -Script $sb.File -Line 7 -Column 5 -Action { })
)

# this will happen when we execute Describe / Context / It / any covered code
& $sb


$breakpoint[0].HitCount
$breakpoint[1].HitCount

# in Exit-CoverageAnalysis
$breakpoint | Remove-PSBreakpoint

[mrboring]

@nohwnd As you pointed out, when the column is specified the breakpoint is not hit. When it's not specified the breakpoint is hit. So, this looks like a PowerShell issue.

Also, I now understand that HitCount is managed by the PowerShell debug engine, and not by Pester.

[nohwnd]

@SteveL-MSFT could you help us understand why the breakpoint on return is not hit, when column is specified, please? Is that by design?

[mrboring]

@nohwnd After a bit of further investigation, based on your sample code:

• If I wrap the return statement in an if block, the breakpoint is not hit.
• If I wrap the return statement in an if block, and add a return value, the breakpoint is hit.

I added the following to your code and updated the Set-Breakpoint commands (line numbers and removed column number).

# Not hit
if ($true) {
    return
}

# Hit
if ($true) {
    return 99
}

[SteveL-MSFT]

Are you sure the column specified points to a valid location in the file? If the line/column is invalid, you don't get an error and no breakpoint is hit.

[nohwnd]

@SteveL-MSFT pretty sure, it hits if the column is not specified, or is before the return. The first breakpoint below is based on the AST extent, the rest are hardcoded, it hits up to 4 but not 5 where return starts.

c:\Users\jajares\Desktop\bp.ps1

Script                          Line Column HitCount
------                          ---- ------ --------
C:\Users\jajares\Desktop\bp.ps1    2      5        0
C:\Users\jajares\Desktop\bp.ps1    2      1        1
C:\Users\jajares\Desktop\bp.ps1    2      2        1
C:\Users\jajares\Desktop\bp.ps1    2      3        1
C:\Users\jajares\Desktop\bp.ps1    2      4        1
C:\Users\jajares\Desktop\bp.ps1    2      5        0
C:\Users\jajares\Desktop\bp.ps1    2      6        0
C:\Users\jajares\Desktop\bp.ps1    2      7        0
C:\Users\jajares\Desktop\bp.ps1    2      8        0
C:\Users\jajares\Desktop\bp.ps1    2      9        0
C:\Users\jajares\Desktop\bp.ps1    2     11        0
C:\Users\jajares\Desktop\bp.ps1    2     12        0
C:\Users\jajares\Desktop\bp.ps1    2     13        0
C:\Users\jajares\Desktop\bp.ps1    2     14        0
C:\Users\jajares\Desktop\bp.ps1    2     15        0
C:\Users\jajares\Desktop\bp.ps1    2     16        0

$sb = {
    return
}

$e = $sb.Ast.FindAll({param($t) $t -is [Management.Automation.Language.ReturnStatementAst] }, $true).Extent 

# in Enter-CoverageAnalysis -> New-CoverageBreakpoint
$breakpoint = @(
    (Set-PSBreakpoint -Script $e.File -Line $e.StartLineNumber -Column $e.StartColumnNumber -Action {}),
    (Set-PSBreakPoint -Script $sb.File -Line 2 -Column 1 -Action { }),
    (Set-PSBreakPoint -Script $sb.File -Line 2 -Column 2 -Action { }),
    (Set-PSBreakPoint -Script $sb.File -Line 2 -Column 3 -Action { }),
    (Set-PSBreakPoint -Script $sb.File -Line 2 -Column 4 -Action { }),
    (Set-PSBreakPoint -Script $sb.File -Line 2 -Column 5 -Action { }),
    (Set-PSBreakPoint -Script $sb.File -Line 2 -Column 6 -Action { }),
    (Set-PSBreakPoint -Script $sb.File -Line 2 -Column 7 -Action { }),
    (Set-PSBreakPoint -Script $sb.File -Line 2 -Column 8 -Action { }),
    (Set-PSBreakPoint -Script $sb.File -Line 2 -Column 9 -Action { }),
    (Set-PSBreakPoint -Script $sb.File -Line 2 -Column 11 -Action { }),
    (Set-PSBreakPoint -Script $sb.File -Line 2 -Column 12 -Action { }),
    (Set-PSBreakPoint -Script $sb.File -Line 2 -Column 13 -Action { }),
    (Set-PSBreakPoint -Script $sb.File -Line 2 -Column 14 -Action { }),
    (Set-PSBreakPoint -Script $sb.File -Line 2 -Column 15 -Action { }),
    (Set-PSBreakPoint -Script $sb.File -Line 2 -Column 16 -Action { })
)

# this will happen when we execute Describe / Context / It / any covered code
& $sb


$breakpoint | select Script, Line, Column, HitCount

# in Exit-CoverageAnalysis
$breakpoint | Remove-PSBreakpoint

[Veretax]

So I was following the example videos for MIcrosoft Virtual Academy, trying to update some of what's written for the latest version of pester, when I did the code-coverage examples, I don't get the missed commands section at all. So I wonder if something changed with how this works for code coverage?

Example here: https://docs.microsoft.com/en-us/shows/testing-powershell-with-pester/code-coverage

[fflaten]

.... I don't get the missed commands section at all. So I wonder if something changed with how this works for code coverage?

Only the summary with percentage etc. is shown by Normal output (default). You need to use Detailed or Diagnostic to get the missed commands section. See answer here: https://github.com/pester/docs/issues/196#issuecomment-1172847657