cypress
cypress copied to clipboard
cypress-io
Be able to run a single test by interacting with it in the Browser
Current behavior:
When you want to run only one spec/context, you must supply .only in code
Desired behavior:
Each test and context should have a deep link which would be interpreted by cypress as the "only". This way, code changes do not need to be made to demonstrate the running of many tests (as currently all tests cannot be run at once). This is how we have used similar test suites, like jasmine, to run a specific suite/spec in the browser without code modifications.
We originally did this and tried this and the workflow doesn't really pan out the way you'd expect. There are all kinds of bizarre edge cases.
It also doesn't really make sense because the entire focus of iterating in the GUI is to focus on a single test, not a group of tests. To give you a UI button to focus on a single test would only be applicable if you were originally running a group of tests - which you shouldn't be doing to begin with.
The other problem is that it can become incredibly difficult if not impossible to track the "only" test.
Imagine you click the "only" in the GUI. I guess we keep track that you clicked on that test by coming up with some identifier, like the index of the test, or maybe its test title, or its full test title including the parents.
The problem is that the test title isn't necessarily unique. When you "re-run" this test we actually have to reload the entire spec file to run it again. There are other tests that may end up having the same name - making it impossible for us to know which one to run. Even if we did find the right one, If you modified the test title or anything above it (a suite, etc) then it would be lost as well. We can't just rerun the callback function of the test because that doesn't actually match the way a real run works and the code may perform differently by us trying to be clever and invoking a callback function more than once.
The second problem is index. That may work but once you modify the code we may lose track of it. What then would we do? Rerun all the tests? Say we can't find the test you put a GUI "only" on?
Also what if you update the code and add a new .only. Do we figure out that you've done that and then remove the "GUI" only and prefer the .only in the code?
Also what happens when you click the GUI "only"? Do we remove all of the other tests as if you've put a real .only in the code? If not, then that's even more state to "keep" to refresh only that single test but leave the others in tact.
We ultimately got rid of this feature because the actual value of it based on the workflow we are going for doesn't make sense. If you could suggest in detail how the implementation of this could work effectively and why this should be the preferred workflow - then we could take another look at this again.
I'd love to run all the tests too, but I guess if I'm showing someone a test suite and they say wow, that's cool, let me see that test again, I'd click and behind the scenes it's essentially like slapping a .only on it
So, yes, the behavior is exactly the same as slapping a .only on it. State is not preserved at all. All tests must work in isolation anyway so this is not what worries me.
As for the spec/link implementation, what I found that there's two ways to go about it (inspired by mocha)
-
When you click on the test it serializes the test name into a string and paste it into the URL. Then it greps for all tests that contain this title. Of all the matching tests it will run each of them
-
In mocha, this stringified test title was passed to the grep command. I found I could easily run a bunch of related tests by just supplying a part of the test name to the grep function
-
Then it easy to run a specific test from typing into the URL - which is quicker and easier than changing code to run a specific test
-
And then finally one last benefit would be when you have the same screen / component across many different functional flows - I could simply include a keyword for this component and now later if I change the component , I can grep and run all the tests with the title in the name.
I think that the grep step would need to parse though all the test cases and pick to run only the test cases which match. As I don't know how Cypress is implemented I'm not sure how it can be accomplished but I'd recommend mocha for the inspiration since I found it quite useful
Cypress is built on top of mocha, we inherit everything that it does. Setting the URL is a possible way to do this. But what happens if you have a grep in the URL, but then you add an .only in your test code. Should the grep be removed? Should we simply say "can't find any tests"?.
Cypress is built on top of mocha, we inherit everything that it does. Setting the URL is a possible way to do this. But what happens if you have a grep in the URL, but then you add an
.onlyin your test code. Should the grep be removed? Should we simply say "can't find any tests"?.
Ideally, the .onlys and .skips in the code would only be used when running all tests in a given file. But if we're running a specific test from a file, ideally, we should be able to run them even if the .only or .skip would normally not include them.
However, if this is not possible because as you say Cypress is built on top of mocha, so be it. Just explain that in the docs and we can work around it. Even so, having this feature would be awesome!
Another possibility is to show all the tests, but show them disabled like the way .skipped test are shown currently, IDK if this is possible.
I agree with Jared and I see the .only as useless and worthy of the trash bin. I need a way to run tests by name/description, a GUI to select which tests to run or exclude, a way to re-run just the failed tests, or just the passing tests. Certainly it can be worked out how to do this.
I have a test spec with 10 tests. The 10th test fails. I fix, it runs all tests again and i have to wait to see if my changes worked. I should be able to run a single test from the browser, just like you can with most unit testing GUI's. The only work around currently is to move the test to the top of the spec, but that is not a good solution. Thank you, please add this feature :-)
Why can’t you run the test using .only when fixing it again? And after it passes remove .only and let all tests run again to make sure nothing else breaks?
Sent from my iPhone
On Dec 26, 2018, at 17:45, sendittokeith [email protected] wrote:
I have a test spec with 10 tests. The 10th test fails. I fix, it runs all tests again and i have to wait to see if my changes worked. I should be able to run a single test from the browser, just like you can with most unit testing GUI's. The only work around currently is to move the test to the top of the spec, but that is not a good solution. Thank you, please add this feature :-)
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.
That is a better work around, but not a solution since you could risk forgetting to remove the .only - but thanks, at least that work around is better :-)
There is eslint plugin that checks “only” and a stand alone utility I wrote called “stop-only” that can catch stray only left in the source code
Sent from my iPhone
On Dec 26, 2018, at 18:03, sendittokeith [email protected] wrote:
That is a better work around, but not a solution since you could risk forgetting to remove the .only - but thanks, at least that work around is better :-)
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.
It would simply be faster and more convenient to be able to click on a single test and have it run. That’s what one expects to happen intuitively.
On Dec 26, 2018, at 5:44 PM, Gleb Bahmutov [email protected] wrote:
There is eslint plugin that checks “only” and a stand alone utility I wrote called “stop-only” that can catch stray only left in the source code
Sent from my iPhone
On Dec 26, 2018, at 18:03, sendittokeith [email protected] wrote:
That is a better work around, but not a solution since you could risk forgetting to remove the .only - but thanks, at least that work around is better :-)
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread. — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cypress-io/cypress/issues/925#issuecomment-450042085, or mute the thread https://github.com/notifications/unsubscribe-auth/AAYGCvB-mqWYvA8IrkpzsPVbQOJy6tAWks5u9An1gaJpZM4QhfgB.
100% putting my vote behind being able to run a single test. If I'm trying to edit a single test, it doesn't make sense to run all of the tests in that spec
@brian-mann @bahmutov , I think what @egucciar suggested above is a near perfect solution! :
- When you click on the test it serializes the test name into a string and paste it into the URL. Then it greps for all tests that contain this title.
You had this question:
Setting the URL is a possible way to do this. But what happens if you have a grep in the URL, but then you add an
.onlyin your test code. Should the grep be removed? Should we simply say "can't find any tests"?.
To answer that, no, don't remove the grep, just tell the user that their test was skipped. Here is a little screen recording of how jest handles running a single grepped test when a .only is applied to another test in that same file:
As you can see, the second time the test is ran with .only applied to another test in that same file, jest just tells us that all of our tests were skipped!
Almost two years late, but:
It also doesn't really make sense because the entire focus of iterating in the GUI is to focus on a single test, not a group of tests. To give you a UI button to focus on a single test would only be applicable if you were originally running a group of tests - which you shouldn't be doing to begin with.
As far as I understand it, the way you run a test from the cypress UI is to click on a spec file and run all the tests in there, right? So does this mean every .spec file should only contain one test? Currently, we have multiple tests (context(...) per .spec file, and if one fails, it just continues with the next one, which isn't really an efficient workflow. Is the solution to split this into multiple .spec files?
This has forced my team into the paradigm where we only have one spec per file and now we are up to 92 files which take over 10 minutes to run in CI. I don't think any frontend testing framework has caught up to junit/eclipse in this regard where you can right click on a single test function and say run/watch test. Jest's watch mode is getting close but having to type in a regex to match a spec name is still pretty cumbersome.
Would love to see the Cypress UI be able to expand a spec file and show all specs within the file, and provide the same run/watch functionality on a single spec from there.
yup, and we kind of have user-space plugins for this
- https://github.com/bahmutov/cypress-select-tests
- https://github.com/bahmutov/cypress-skip-and-only-ui
But running a single test as part of the Cypress core would be ideal feature
On Wed, Apr 24, 2019 at 9:12 AM Matt [email protected] wrote:
This has forced my team into the paradigm where we only have one spec per file and now we are up to 92 files which take over 10 minutes to run in CI. I don't think any frontend testing framework has caught up to junit/eclipse in this regard where you can right click on a single test function and say run/watch test. Jest's watch mode is getting close but having to type in a regex to match a spec name is still pretty cumbersome.
Would love to see the Cypress UI be able to expand a spec file and show all specs within the file, and provide the same run/watch functionality on a single spec from there.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cypress-io/cypress/issues/925#issuecomment-486222921, or mute the thread https://github.com/notifications/unsubscribe-auth/AAQ4BJSKCPHKPQKI2TWOUHTPSBMDRANCNFSM4EEF7AAQ .
-- Dr. Gleb Bahmutov, PhD
Schedule video chat / phone call / meeting with me via https://calendly.com/bahmutov [email protected] @bahmutov https://twitter.com/@bahmutov https://glebbahmutov.com/ https://glebbahmutov.com/blog https://github.com/bahmutov
Wow, I didn't know the plugin API could modify the UI. The docs certainly don't suggest it. It would also be cool to find your plugins on https://docs.cypress.io/plugins/index.html @bahmutov
The lack of this feature is very painful when you have to debug why individual tests are failing.
I'm currently running a single test by renaming the it declarations to xit as per mocha API and all seems to be ok, given that our tests are self contained.
it.only(....) also works great.
@bahmutov I'm interested in helping implement this feature. Do you know if any work has been done on it yet? Could you give me some direction on a high level of what would need to be done? Is this even something that has officially been approved?
Thanks for the amazing product!
Another use case is a test might be flakey in frequently, and one might want to just retry or re-run the failed and then everything else, as per numerous junit runners (java).
Looking for the same feature and sad that https://github.com/bahmutov/cypress-skip-and-only-ui appears to be broken.
In case it helps anyone, I found a simple solution using https://github.com/cypress-io/cypress-grep. Since it exposes the ability to grep at runtime using Cypress.grep(), it wasn't hard to write an after hook that adds buttons next to each test and suite that call it with the name of the test/suite:
Super, this is a good solution
Sent from my iPhone
On Mar 3, 2022, at 14:12, jcupps @.***> wrote:
Looking for the same feature and sad that https://github.com/bahmutov/cypress-skip-and-only-ui appears to be broken.
In case it helps anyone, I found a simple solution using https://github.com/cypress-io/cypress-grep. Since it exposes the ability to grep at runtime using Cypress.grep(), it wasn't hard to write an after hook that adds buttons next to each test and suite that call it with the name of the test/suite:
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.
@tnrich Feel free to use my quick-and-dirty solution: https://github.com/jcupps/cypress-grep-buttons
Still waiting for a built in solution for this. Please add one. Love the new debug interface but it could also use a "Run single test" option.