cypress
cypress copied to clipboard
cypress-io
Allow to rerun only failed tests directly from the Test Runner UI
Current behavior:
In the test runner, there is no option to only rerun failed tests. After investigating and fixing a failed test, I'd like the watcher to only rerun failed tests.
Desired behavior:
Add a rerun "Rerun failed tests only" checkbox to the tests runner UI.
Some similiar requests:
- https://github.com/cypress-io/cypress/issues/1313
- https://github.com/cypress-io/cypress/issues/1599
@jennifer-shehane and @Superkunair you both refer to issues and solutions to handle flaky tests by rerunning them. That's not what I'm looking for. I'm looking for a feature in the Test Runner UI.
The use case is:
- I change some code and then run the tests using the Test Runner UI.
- Running them takes 10 minutes and 3 tests fail.
- For the next several minutes I'll focus on these 3 tests, changing code and rerunning them many times.
- While I do this, I want the Test Runner UI to only watch/rerun these 3 tests, until they pass.
- When I'm done fixing the issue, I can rerun the whole batch before I commit and push.
So this request is really for an improved developer experience while investigating issues, not as a solution to handle flaky tests.
Yes, just tagging similar requests - this is a new kind of request which is why the issue was left open.
There's a plugin that can execute your failed test cypress-plugin-retries
That is for re-running during execution to account for network issues? We want a button in the runner marked "re-run failed tests only) to allow for debugging. Don't confuse the issue.
@jennifer-shehane and @Superkunair you both refer to issues and solutions to handle flaky tests by rerunning them. That's not what I'm looking for. I'm looking for a feature in the Test Runner UI.
The use case is:
- I change some code and then run the tests using the Test Runner UI.
- Running them takes 10 minutes and 3 tests fail.
- For the next several minutes I'll focus on these 3 tests, changing code and rerunning them many times.
- While I do this, I want the Test Runner UI to only watch/rerun these 3 tests, until they pass.
- When I'm done fixing the issue, I can rerun the whole batch before I commit and push.
So this request is really for an improved developer experience while investigating issues, not as a solution to handle flaky tests.
THIS! It is a really super simple and basic feature. Every IDE has it. Only Cypress runner fails to provide such a simple thing!
Some similiar requests:
- #1313
- #1599
Neither of those issues relates to this issue in any way.
I can agree with @DavidGWheeler, we're looking forward to see this feature too. Currently there isn't any way to rerun just failed test after fixing it.
I've found two options:
- Use
it.only(...)in your tests - https://github.com/bahmutov/cypress-skip-and-only-ui
This is a must for test-driven development.
For me jest in watch mode (via vue-cli) has the ideal behaviour, it goes one step further and only re-runs the tests affected by code changes since the last time they ran (so that if there are 5 failing tests and I modify 1 of them only that one will re-run).
However I don't want to confuse the issue and as the OP suggested, an option for only re-running failing tests is a perfect first step.
While I don’t agree with @DavidGWheeler 's statement about it being a feature that is both simple and easy (if it’s so easy, please submit a PR), I think this feature is really essential to the workflows on projects I’m working on.
I was really excited about the Cypress 5.0 when I read about the retry ability - but this is really not what I had in mind. I think having the ability to re-run a single failed test, or multiple failed tests in a row, would really help.
On a test suite of 130 tests that take 20 minutes to run, seeing 13 failed tests (completely separated) and re-running them is a nightmare once I changed the behavior to fix them.
In some other testing frameworks I've used they have this feature built-in (RSpec, Cucumber)
When the test fails it will create a file (eg. .rspec_retry) with filenames with line of code number of failing tests, and the next test run will focus only these tests. You don't have to manually add focus like .only to any failing tests manually. When all tests are passing, the file will be deleted.
This feature is not about mitigating flaky tests, but for TDD or auto-focusing test for fixing purpose.
Hi ! As I filled in #17803, it would nice to have TDD mode : GREEN, RED, REFACTOR which will make Cypress behaves differently depending to the mode.
Is there any update on this issue? I've seen a number of tickets related to this but it's not clear if this functionality is going to be added or where to track the progress.
@jennifer-shehane Sorry to "me too" a feature request, but I think jest has made this an expected feature in a test runner (and much more important for integration test suites that can take multiple minutes to run in the average case scenario). Since it has been open for two years and other tickets are being closed as duplicates, is this currently on the roadmap?
An update on this or a solution - it is soul destroying having to re-watch the entire set of tests to get to the failed ones for each iteration whereby you are trying to fix the problem
For people that center their work on TDD, this feature would be really helpful.
An update on this or a solution - it is soul destroying having to re-watch the entire set of tests to get to the failed ones for each iteration whereby you are trying to fix the problem
Agreed, this is extremely painful for a TDD workflow.
This request should not be limited to "failed" tests. While iteratively writing a test, it is a must to be able to tweak a single test then rerun it, whether it failed or not.
@adamvanaken For that behavior you can add it.only or describe.only to focus on a single test.
@danhooper that's fine, but when running several specs (not possible on cypress 10 anymore), it would be very helpful to re-run only the failed ones. This way you shorten your list for each test fixed until you have all fixed while avoiding rerunning tests that already passed
@Carniatto Agreed. My response was only to help with when you are developing tests.
I very much want the ability to rerun failed tests.
did you find any solution for this brother, even I am looking for this as cypress test cases are independent of each other and the data being used by them is independent so yes even we are with the same requirement now to only rerun failed test cases (but not retries to test flaky one)
