Retry failed assertions

[amannn]

Is your feature request related to a problem? Please describe.

Due to variability, Lighthouse CI may sometimes report a lower performance score that causes an assertion to fail, whereas for most runs, the assertion passes. Since the Lighthouse test takes rather long, I've configured numberOfRuns: 1 for my CI pipeline. However, if, and ideally only if, a performance assertion fails, it would be helpful if the test could be re-run for a given URL.

Increasing numberOfRuns isn't quite the same thing, as it will run multiple times also when the first test passes.

Describe the solution you'd like

An option to retry the assertions for a given page if the the first run fails.

Describe alternatives you've considered

Wrapping the entire LHCI run to re-run if it fails (this needlessly tests URLs again that have already passed).

[amannn]

@paulirish Is there someone at Google who's taking care of this project? Would be really helpful to see this implemented, with numberOfRuns: 1 the CI results can unfortunately be quite flaky.

[amannn]

@connorjclark Any chance this could be considered?

[paulirish]

Hey Jan. We'd be open to a pull request for this but we won't be able to write this feature ourselves anytime soon.

[amannn]

Hey Paul, thanks for getting back to me!

I see. I'm currently also a bit short on time, but I'll keep this in mind potentially for the future!