Pester Provide machine-friendly, streaming output format

Checklist

[X] Feature request has a meaningful title
[X] I have searched the existing issues. See all issues
[X] I have tested using the latest version of Pester. See Installation and update guide.

Summary of the feature request

Our ES testing infrastructure has its own reporting mechanism, and we've previously integrated with many other test frameworks, including GoogleTest, Catch2, Boost.Test, VSTest / xUnit.Net / NUnit / MSTest. Most of these frameworks provide a built-in output format that is streamed (provides the data as the tests are running) as well as programmatically-processible (XML written to stdout for Catch2 and Boost.Test, URL-like lines written to a socket for GoogleTest). Could Pester provide such an output format as well?

This would provide a mechanism that could be used to adapt Pester test results to any other format, while the Pester tests are running. Runners such as VS Code might also make use of this format as a stable way of presenting results, rather than needing the currently-non-public add-in mechanism.

How should it work?

Invoke-Pester (or the configuration object) provides a way to specify that the output mode is machine-friendly (rather than the default of human-friendly). Perhaps a new Output.Mode on PesterConfiguration, with values of 'Human' (default) and 'Machine' (new), with machine output looking something like this (inspired by the GoogleTest machine-friendly output format):

pester_streaming_protocol_version=1.0
event=PesterStart&version=6.0.0
event=DiscoveryStart
event=Discovered&path=C:\t\Planets\Get-Planet.Tests.ps1
event=DiscoveryEnd
event=RunStart&path=C:\t\Planets\Get-Planet.Tests.ps1
event=DescribeStart&name=Get-Planet
event=ItStart&name=Given%20no%20parameters,%20it%20lists%20all%208%20planets
event=ItEnd&result=fail&duration=40ms&message=The%20term...&file=C:\t\Planets\Get-Planet.Tests.ps1&line=7
event=DescribeEnd
event=RunEnd

Nov 20 '24 21:11 davidmatson

Other options that could work for the same scenario:

XML to stdout: Parsing streaming XML (no DOM, as the file isn't fully written yet) is harder than the format above, but Catch2 and Boost.Test use this mechanism, and it could work.

Add-in: This requires getting the caller involved more in the execution environment than passing a switch and consuming a programmatic format from stdout. We had to use this approach for VSTest, and it was much less flexible for execution environment integration.

Nov 20 '24 21:11 davidmatson

Related to #550 and #447 - this might work for both of those cases as well as others.

Nov 20 '24 22:11 davidmatson

@nohwnd - thoughts on this feature? We'd be interested in working on an implementation draft, if there's agreement that this would be a good idea.

Dec 03 '24 21:12 davidmatson

I can't promise anything right now, I work on v6 when I can, and my time is limited.

This does not look like a feature that would be usable to everyone. Do you see some uses for this that would allow us to support some new exciting thing, rather than just your own reporting mechanism? e.g better ide integration in vscode.

Dec 12 '24 20:12 nohwnd

Also related to #2021 so you could add or replace with custom output.

Dec 13 '24 11:12 fflaten

@nohwnd - yes, I'd imagine VSCode integration could use this rather than the currently-unsupported hook that it's currently using to show progress in the IDE as tests are executing. Thoughts?

Dec 19 '24 20:12 davidmatson

@nohwnd Could you emit an Event at the end of Invoke-Block that has $block as the data? I'm looking to try and do something similar to this xUnit Otel instrumentation. That would allow a user access to subscribe to the raw object and should provide some useful functionality for IDEs - though I don't know how the VSCode Pester integrations work currently.

Jan 08 '25 17:01 cdhunt

I did a crude POC.

Test

Describe "Pester Otel integration" {
    It "fake test" {
        $true | Should -BeTrue
        New-Event -SourceIdentifier Pester -Sender $____Pester.CurrentBlock.Name -MessageData $____Pester.CurrentBlock
    }
}

Data

$Events = Get-Event -SourceIdentifier Pester
$Events[-1].MessageData | select Name, Passed

Name                    Passed
----                    ------
Pester Otel integration   True

Jan 08 '25 18:01 cdhunt

The current vscode integration that Justin did registers a plugin into the standard plugin subsystem we have. Can’t share details now, but will try to do some coding / maintenance on pester in the evening, in an hour or so.

-j

Od: Chris Hunt @.> Odesláno: Wednesday, January 8, 2025 7:01:16 PM Komu: pester/Pester @.> Kopie: Comment @.>; Subscribed @.> Předmět: Re: [pester/Pester] Provide machine-friendly, streaming output format (Issue #2581)

I did a crude POC.

Test

Describe "Pester Otel integration" { It "fake test" { $true | Should -BeTrue New-Event -SourceIdentifier Pester -Sender $____Pester.CurrentBlock.Name -MessageData $____Pester.CurrentBlock } }

Data

$Events = Get-Event -SourceIdentifier Pester $Events[-1].MessageData | select Name, Passed

Name Passed

Pester Otel integration True

— Reply to this email directly, view it on GitHubhttps://github.com/pester/Pester/issues/2581#issuecomment-2578299025 or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABLYLYOF3VAYYNJT6NXQTOT2JVRW3BFKMF2HI4TJMJ2XIZLTSSBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVEZDAMJYGYZTKOJXURXGC3LFVFUGC427NRQWEZLMVRZXKYTKMVRXIX3UPFYGLLCJONZXKZKDN5WW2ZLOOSTHI33QNFRXHE4CUR2HS4DFVJZGK4DPONUXI33SPGSXMYLMOVS2OMJSGEZTINZUQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSGY3TOMRRGAYDKMECUR2HS4DFUVWGCYTFNSSXMYLMOVS2SMRQGE4DMMZVHE32O5DSNFTWOZLSUZRXEZLBORSQ. You are receiving this email because you commented on the thread.

Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Jan 08 '25 18:01 nohwnd

There's a plugin subsystem?

Jan 08 '25 18:01 cdhunt

Yes internaly most of pester functionality is implemeted as plugins that subscribe to “events” coming from the main engine. So we don’t mix up the order of executing them. Can handle failures in consistent way and handle scoping / isolation correctly. New-pesterPlugin is imho how the function that produces plugin object is called, if you find that it should be more obvios what is going on.

-j

Od: Chris Hunt @.> Odesláno: Wednesday, January 8, 2025 7:34:23 PM Komu: pester/Pester @.> Kopie: Comment @.>; Subscribed @.> Předmět: Re: [pester/Pester] Provide machine-friendly, streaming output format (Issue #2581)

There's a plugin subsystem?

— Reply to this email directly, view it on GitHubhttps://github.com/pester/Pester/issues/2581#issuecomment-2578357559 or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABLYLYJFOYL547MLN6SHKCL2JVVS7BFKMF2HI4TJMJ2XIZLTSSBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVEZDAMJYGYZTKOJXURXGC3LFVFUGC427NRQWEZLMVRZXKYTKMVRXIX3UPFYGLLCJONZXKZKDN5WW2ZLOOSTHI33QNFRXHE4CUR2HS4DFVJZGK4DPONUXI33SPGSXMYLMOVS2OMJSGEZTINZUQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSGY3TOMRRGAYDKMECUR2HS4DFUVWGCYTFNSSXMYLMOVS2SMRQGE4DMMZVHE32O5DSNFTWOZLSUZRXEZLBORSQ. You are receiving this email because you commented on the thread.

Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Jan 08 '25 18:01 nohwnd

Register callbacks, rather that subscribe to events.

-j

Od: Jakub Jareš @.> Odesláno: Wednesday, January 8, 2025 7:37:26 PM Komu: @. @.>; pester/Pester @.> Kopie: Comment @.>; Subscribed @.> Předmět: Re: [pester/Pester] Provide machine-friendly, streaming output format (Issue #2581)

Yes internaly most of pester functionality is implemeted as plugins that subscribe to “events” coming from the main engine. So we don’t mix up the order of executing them. Can handle failures in consistent way and handle scoping / isolation correctly. New-pesterPlugin is imho how the function that produces plugin object is called, if you find that it should be more obvios what is going on.

-j

Od: Chris Hunt @.> Odesláno: Wednesday, January 8, 2025 7:34:23 PM Komu: pester/Pester @.> Kopie: Comment @.>; Subscribed @.> Předmět: Re: [pester/Pester] Provide machine-friendly, streaming output format (Issue #2581)

There's a plugin subsystem?

— Reply to this email directly, view it on GitHubhttps://github.com/pester/Pester/issues/2581#issuecomment-2578357559 or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABLYLYJFOYL547MLN6SHKCL2JVVS7BFKMF2HI4TJMJ2XIZLTSSBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVEZDAMJYGYZTKOJXURXGC3LFVFUGC427NRQWEZLMVRZXKYTKMVRXIX3UPFYGLLCJONZXKZKDN5WW2ZLOOSTHI33QNFRXHE4CUR2HS4DFVJZGK4DPONUXI33SPGSXMYLMOVS2OMJSGEZTINZUQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSGY3TOMRRGAYDKMECUR2HS4DFUVWGCYTFNSSXMYLMOVS2SMRQGE4DMMZVHE32O5DSNFTWOZLSUZRXEZLBORSQ. You are receiving this email because you commented on the thread.

Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Jan 08 '25 18:01 nohwnd

That would be simple enough to adapt for my purposes.

Jan 08 '25 18:01 cdhunt

This is the change we did for justin to allow adding external plugins: https://github.com/pester/Pester/pull/2020

-j

Od: Chris Hunt @.> Odesláno: Wednesday, January 8, 2025 7:56:10 PM Komu: pester/Pester @.> Kopie: Comment @.>; Subscribed @.> Předmět: Re: [pester/Pester] Provide machine-friendly, streaming output format (Issue #2581)

That would be simple enough to adapt for my purposes.

— Reply to this email directly, view it on GitHubhttps://github.com/pester/Pester/issues/2581#issuecomment-2578396432 or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABLYLYNXPE6QKTUNJASBWHD2JVYEVBFKMF2HI4TJMJ2XIZLTSSBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVEZDAMJYGYZTKOJXURXGC3LFVFUGC427NRQWEZLMVRZXKYTKMVRXIX3UPFYGLLCJONZXKZKDN5WW2ZLOOSTHI33QNFRXHE4CUR2HS4DFVJZGK4DPONUXI33SPGSXMYLMOVS2OMJSGEZTINZUQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSGY3TOMRRGAYDKMECUR2HS4DFUVWGCYTFNSSXMYLMOVS2SMRQGE4DMMZVHE32O5DSNFTWOZLSUZRXEZLBORSQ. You are receiving this email because you commented on the thread.

Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Jan 08 '25 18:01 nohwnd