Reporting Test Execution Time and Flakiness

[kms254]

What product do you want to improve? CodeCov Web Application and CLI Uploader

Is your feature request related to a problem? Please describe. We are currently a Pro Customer who recently onboarded CodeCov. One of the key metrics we monitor is testing time and flakiness. Using multiple tools to monitor test status is inefficient and adds cognitive load.

Describe the solution you'd like It would be great if CodeCov could be our single pane of glass regarding testing. If in the future we can include test execution time and flakiness that would be a value add. Especially as we build out historical data over time so we can see long-term trends.

Describe alternatives you've considered Currently, we use DataDog(dd_trace) to capture and visualize testing metrics regarding flakiness and execution time. The bifurcation of testing data is cumbersome and I would love for it to all be in one tool.

[rohan-at-sentry]

@kms254 we're actually doing some discovery around this currently. Would you be open to share thoughts around what you'd like to see (and what sort of tests you consider as flaky)

[kms254]

@rohan-at-sentry Sure thing. We primarily look at test execution time overall and want to monitor the trends. When digging into long-running tests, individual test run time is very important but is only looked at if there is a problem. Our goal is to focus on fast, efficient testing without having the overhead of having to tune our test cases. We have some integration tests that, depending upon the test state, can be flakey. Knowing this rate gives us an idea of where to invest the time to remove the friction and have more reliable testing. If need be I'm happy to get on a screen share with you to show you what we are doing.

[rohan-at-sentry]

I'd love to see what you're doing here. Would you mind booking a time that works for you via this link? https://calendar.app.google/jpMWdByFsDKFq72t9

[covecod[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[Adal3n3]

We would love your insight about how would you prefer to receive failed tests report on PR? https://github.com/codecov/feedback/discussions/189

Thanks a ton!

[rohan-at-sentry]

@kms254 , we're currently building out the ability to detect flaky tests and report them on the PR. We'd love the opportunity to pick your brain on how you're doing this currently to see how we can help improve that process for you. If this is of interested, please respond back and I can work with you to schedule some time