Allow reporters to write down data before panicking on timeout

[lukas-bednar]

Hello,

When I set a timeout for entire test suite using --timeout parameter, the test eventually fail (panic) on given timeout - that is OK. The issue is that I lose all data collected by reporters (junit reporter in my particular case). My expectation is that framework can generate partial report, before panicking entire run.

Is it possible to add additional interface to the reporters to write down collected data before exiting?

[InbarRose]

+1

[onsi]

hmm - i might need to take a look at this but i think the fix might be a bit tricky. the suite-level timeout parameter was always intended as a last-ditch failsafe - the expectation being that individual tests that test asynchronous behavior are written with appropriate granular timeouts (e.g. with Gomega's Eventually). My instinct is that adjusting the --timeout parameter down is a bit of a smell...

I appreciate that doesn't actually help solve your problem. But I'd love to hear if it spurs any thoughts on how you're approaching the problem?

Also - if there's more detail around the underlying problem you're trying to solve with a (presumably) shorter timeout I'd love to hear it as that might help me offer alternative solutions.

For now, I'm going to add this to the v2 list. v2 is under development though it's been slow going...

[mat007]

I have been looking for something like this because we use a custom reporter to send the test results to an elasticsearch instance. We have dozens of separate test runs occurring all day long and we use elasticsearch to query for failures and such. When the general go test timeout hits, we miss that test run in the data.

I’ll probably try to catch the panic and force send the data with the reporter now to see how it goes.

Thanks!

[onsi]

I can confirm I'm going to address this in v2. However, I'm also considering removing support for custom reports as well in lieu of supporting specific output formats that users can post-process. I'm going to maintain the JUnit support and introduce a new JSON format that has all the information that currently goes to custom reporters.

@mat007 I'd like to understand how disruptive that would be for your elasticsearch reporter. Could you (relatively) easily retool to have a post-run hook that sends the JSON report to elasticsearch instead?

[mat007]

@onsi yes I think based on your description of this feature it looks fine. Thanks!

[onsi]

Glad to hear it, thanks @mat007 !

[onsi]

This is now resolved in V2. Ginkgo now manages its own timeout and triggers a test-wide interrupt that gracefully unwinds any running tests and calls reporters appropriately. V2 is in beta and I'd love to see folks try it and give me feedback. Docs for using the beta are here.

[mat007]

I’ve switched our e2e tests to V2 with only minor code rewrite. It’s working fine, including in the timeout case I was describing above.

Thanks @onsi !

[onsi]

that's great to hear, thanks!