AzureDevOpsExtension
AzureDevOpsExtension copied to clipboard
pester
Pester returns errors even when no tests fail
Where are you running it?
- Azure DevOps Service (VSTS)
Version of Extension/Task
10.0.5
Expected behaviour and actual behaviour
Expected: Pester task passes without error when no tests fail
Actual: Pester task fails with error even though all tests pass
Steps to reproduce the problem
Here's a link to a public project where this issue can be repro'd
https://dev.azure.com/cfarrelly/test/_build/results?buildId=4&view=logs&j=275f1d19-1bd8-5591-b06b-07d489ea915a&t=f969aa47-2434-555b-2817-6587663ed32a
I thought originally it may be down to tags and skipping tests but I created a simple repro here with 1 test to run and the test runner fails with 'Pester returned errors', appears to be an issue when checking the results here
https://github.com/pester/AzureDevOpsExtension/blob/e2141f28ee869c892dfc91e3191145da14f125d3/Extension/PesterTask/PesterV10/Pester.ps1#L145
Thanks,
Colin.
Thanks for reporting this. I'll have a look at what's going on here and see if I can fix it this week.
Just got bitten by this one too: was starting to add some tests and couldn't figure out why they were passing on my local machine and even showing as passing in the output logs but then the task fails with:
D:\a\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester.ps1 : Pester returned errors
At line:1 char:1
D:\a\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester.ps1 : Pester returned errors
At line:1 char:1
+ d:\a\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Pester.ps1
+ d:\a\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Pester.ps1
Thanks for the extra reports. I'll bump this up my list and see if I can get it done tonight or earlier.
Just hit this bug as well...
2020-06-03T12:15:39.5996504Z Tests Passed: 3,
2020-06-03T12:15:39.6033071Z Failed: 0,
2020-06-03T12:15:39.6070344Z Skipped: 0
2020-06-03T12:15:39.6150264Z NotRun: 0
2020-06-03T12:15:39.6180214Z
2020-06-03T12:15:39.6181136Z
2020-06-03T12:15:40.2186133Z ##[error]C:\Build\CI-Agent2\_work\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester.ps1 : Pester returned errors
At line:1 char:1
2020-06-03T12:15:40.2198914Z ##[error]C:\Build\CI-Agent2\_work\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester.ps1 : Pester returned errors
At line:1 char:1
2020-06-03T12:15:40.2217307Z ##[error]+ C:\Build\CI-Agent2\_work\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3 ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Pester.ps1
2020-06-03T12:15:40.2220096Z ##[error]+ C:\Build\CI-Agent2\_work\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3 ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Pester.ps1
2020-06-03T12:15:40.2716171Z Pester Script finished
2020-06-03T12:15:40.2875525Z ##[section]Finishing: Pester Test Runner
I too faced the same error and waiting for the solution. Please find below for the raw error log report.
Also the test result is showing as passed even though I intentionally placed the code to fail the test.
2020-06-04T10:22:55.0696725Z ##[error]D:\a_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester.ps1 : Pester returned errors At line:1 char:1
- d:\a_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester ...
2020-06-04T10:22:55.0709967Z ##[error]D:\a_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester.ps1 : Pester returned errors At line:1 char:1
- d:\a_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester ...
2020-06-04T10:22:55.0729293Z ##[error]+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Pester.ps1
2020-06-04T10:22:55.0731667Z ##[error]+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Pester.ps1
2020-06-04T10:22:55.1061073Z Pester Script finished 2020-06-04T10:22:55.1140399Z ##[section]Finishing: Pester Test Runner
Thanks for the extra reports, I'm working on a fix but I also want to improve my testing in the pipeline to make sure I catch more of these issues before they hit users. It's going to take me a few hours of work to do that so I'm hoping to get it done this weekend.
This should be fixed now in 10.1.x, took a bit longer than expected as my testing pipeline needed work and then wasn't quite doing what it should have.
Hi @ChrisLGardner Thanks for resolving the errors. Still i am able to see the issue as below "test result is showing as passed even though I intentionally placed the code to fail the test"
Can you please look into this too.
Thanks Indra
Hi @ChrisLGardner Thanks for resolving the errors. Still i am able to see the issue as below "test result is showing as passed even though I intentionally placed the code to fail the test"
Can you please look into this too.
Thanks Indra
I saw this a few times when I was fixing the task. What version is it trying to run? If you look in the logs for the task it'll give you an exact version number at the top.
Task : Pester Test Runner Description : Run Pester tests by either installing the latest version of Pester at run time (if possible) or using the version shipped with the task (5.0.0) Version : 10.1.12 Author : Pester Help : Version: #{Build.BuildNumber}#. More Information
It looks like the Pester5 result object has changed significantly since v4 and no longer has a failedcount property. Helper command ConvertTo-Pester4Result is available to transform a v5 result object to v4 format.
Also, what do we do if we need to use Pester v4? Since v10 of the task with this fix doesn't support it.
I can't upgrade all my tests yet because of the bug in Pester v5 where variables in BeforeAll are only set on the second run of the tests.
You can use v9 of the task as that supports v4 of Pester.
I didn't get a chance to look at this at the weekend but I'll try to find some time this week.
You can use v9 of the task as that supports v4 of Pester.
I didn't get a chance to look at this at the weekend but I'll try to find some time this week.
I mean once the fix is put out, unless you're planning to update v9 of the task as well (I'm not familiar with the task publishing process so not sure if that is possible).
You can use v9 of the task as that supports v4 of Pester. I didn't get a chance to look at this at the weekend but I'll try to find some time this week.
I mean once the fix is put out, unless you're planning to update v9 of the task as well (I'm not familiar with the task publishing process so not sure if that is possible).
My plan is to maintain v9 of the task as long as Pester 4 is in use, it hopefully doesn't need many fixes as it's had plenty of time to get those so far but anything we find in v10 that is an issue in v9 will get fixed in both.
I've added another fix that should hopefully help with this issue if people are still seeing problems.