cypress
cypress copied to clipboard
cypress-io
Ability to run spec files in a specific order
How can I run all tests in a order without rename the files using a number to determine the order to execute?
There is a another way that can I run the tests in a custom order?
For now, I’m renaming the files using the prefix 1_xxx.js, 2_xxx.js
No, you shouldn't need to run tests in any specific order. That is usually indicative of a testing anti pattern, whereby you are relying on state being built up between all the tests.
Can you explain your situation / use case?
For example: before my other tests, i need create a user. But what i really want is https://docs.cypress.io/docs/configuration#section-global
// ***********************************************************
// This example support/index.js is processed and
// loaded automatically before your other test files.
//
// This is a great place to put global configuration and
// behavior that modifies Cypress.
//
// You can change the location of this file or turn off
// automatically serving support files with the
// 'supportFile' configuration option.
//
// You can read more here:
// https://on.cypress.io/guides/configuration#section-global
// ***********************************************************
// Import commands.js and defaults.js
// using ES2015 syntax:
import "./commands"
import "./defaults"
// Alternatively you can use CommonJS syntax:
// require("./commands")
// require("./defaults")
TXS @brian-mann !
Ordering tests could be convenient if you want to run the often failing tests first. It saves you some time in CI. Another use case is when you have a lot of scenarios and you want to organize them in a way that makes more sense than alphabetically.
@brian-mann I have a use case for why it is really important to be able to run tests in order. We are using Cypress for functional live tests.
When running live tests we are dealing with real customer data that is somewhat dependent on state, in order for us to run our tests we need to hit real endpoints and perform actual CRUD operations. If we cannot control what test packages are called in an order the tests will fail since clearing out and creating data in order is a big part of live testing.
I believe this is somewhat related to this issue: https://github.com/cypress-io/cypress/issues/263
Right now we are resorting to labelling our test folders 01_Users, 02_Invoices, etc... to control the test ordering which just makes me very sad.
As @digitaldavenyc points out full blown web apps often test complex system interactions.
Should every test be able to run independently and in a vacuum? Sure. In theory. Real world and theoretical science do not always get along. It may be anti-pattern but coloring 100% within "the pattern" lines means my test scripts take 27 hours to complete instead of 27 minutes.
Use Case--
My app allows users to add locations to a data set on the back end. On the front end it renders a map of locations.
Front-End test: Make sure 10 locations are returned from Charleston SC
This is heavily dependent on the data set that includes two primary elements: location data + user-set options such as "default radius to be searched".
I could write the test to first pull the options data (default: 5 mile radius), THEN pull all location data (5k entries), THEN let the script loop through all 5,000 to calculate the distance to create the baseline of valid results, THEN run the front-end query and make sure it returns the same list my test calculated.
I've now created another point of failure as my test script code is more complex and prone to errors. Not too mention it takes a LOT longer to run by not being able to make data assumptions.
OR
I could write a test that ASSUMES my "load_5k_locations_spec.js" has executed and passed. The test is now pull the option data, load a "valid_5_miles_from_charleston.json" fixture, run the front-end query and compare it against the displayed location divs.
An order of magnitude faster AND far less complex test script code.
NOW... take the above and run the test for five different radius options.
I'd rather pull a valid_5_mile.json, valid_10_mile.json, etc. and compare against an assumed set of data that can ONLY be a valid assumption if I am certain my "load_5k_locations_spec.js" ran BEFORE all my "5miles_from_charleston.js" and "10miles_from_charleston.js" scripts ran.
Bonus Points: Have a Cy.passed( "load_5k_locations_spec" ) command the returns true|false if the specified test passed on the prior run -- makes it easy to completely skip a test if a prior test run failed.
No - not perfect "by the pattern" rulebook implementations, but in the real world people have deadlines. Making tools that are malleable enough to meet user's needs versus doing things "strictly by the book" is what makes them powerful.
I'm fairly certain chain saws are not designed to be used to make ice sculptures - but people do it. If you try to do the same and cut your arm off because you don't have the skills to use that tool in a manner it was not intended, consider it a "learning curve".
I strongly suggest NOT going down the path "real world requires running spec files in order". It is just JavaScript and you can easily set state before each test the way you want, without relying on the previous test. In fact we are working on automatic test balancing so different specs will be running on different CI machines and in different order (if you use --parallel flag). So any test order will not be guaranteed.
Instead, split tests into functions and just call these functions to set everything up.
I agree with @cristopher-rodrigues that running tests in order has major advantages - speed wise. You can save time but not reloading the login page(or doing requests for i) each time
It is, indeed, a huge pain to run e2e tests with an SSO redirect (which can't be removed) if you cannot make sure your login procedure runs as the first test to set a valid token. Otherwise, you have to write logic in every single test file to handle that, which seems silly.
@RadhikaVytla you could run the login_spec first as an isolated job (CI), and depend all other test specs on that job being successful. Could save you some time if login isn't working.
@araymer can you set the token through an auth api call? If so, just use a beforeEach (write it once) and this way each test can use it's own separate auth token
@araymer You can also move this to the support file within a beforeEach as described here: https://on.cypress.io/writing-and-organizing-tests#Support-file
Being able to specify the order could be very useful when testing a process composed by several steps. I'd like each step to be a standalone test, but they need to be executed in a specified order. Currently I can only follow the advice that naming them with 01, 02..
Hi, All! I have the same problem. I want to run tests in a specific order: 1.SigUp, 2.Login, 3.ShareTool, 4.Publication and then 5.Account.
However tests run like on this screenshot:
First "ShareTool.feature" runs and then next tests according to structure in cypress project:
Just found an easy hack, add a number before your file or folder and it will run those in that order.
For example, @id-dan you can do something like 01-SignUp/01-SignUp.feature/, 01-SignUp/02-Login.feature/.....05-Account/... and so on.
Cypress should pick up the files in that order and run them.
This would be a great feature. I currently have two test files that take a lot longer to run and are "more flakey" than others resulting in the occasionally fail for non legitimate reasons.
Due to the file names (upgrade_page.test.ts & settings_page.test.ts) both these files run at the end of a 15+min e2e suite. I could save time and money with my CI provider if I were able to run these tests first so I can bail out of remaining tests if they fail
Any news on this @brian-mann ? Cy best practice for each test to run standalone is excellent and perfectly architected, but there's definitely an added convenience to being able to order tests, and I'm curious if you and your guru coders are considering it? Thanks!
To reiterate, this feature would be really useful in order to run often-failing/fragile tests first, thereby ensuring that failures are flagged as quickly as possible. Some workarounds for this have been described in this issue thread (such as numbering tests in intended order), but this isn't really a solution, as it relies on a cumbersome manual renaming of test files. If test ordering could be specified by some configuration file, this file could be generated and updated by subsequent test runs, putting frequently-failing tests first.
I would love to get an update on whether this feature is being considered.
No, you shouldn't need to run tests in any specific order. That is usually indicative of a testing anti pattern
That's correct for unit tests, however integration and end to end tests are a different beast. I appreciate the additional power cypress gives us for injecting state directly into the page and the cy.request feature to avoid having to do things like ordered tests. But sometimes you just need ordered tests and the whole "tests should be able to run in any order" is simply a cargo cult mindset without considering the context.
I don't think "tests should be able to run in any order" is a cargo cult. Even in e2e, randomizing test order (which cypress doesn't support yet, though) can reveal bugs in DB or anywhere where state is preserved.
That being said, I too would like this feature implemented simply from the standpoint of debugging failing tests, as many have mentioned before, and not just in CI. For example, the issue I've posted earlier could be reproduced only by running two spec files in consecutive order, and to isolate this I had to manually remove all other spec files so that only the two spec failing files remained, and I could iterate much more quickly in debugging (instead of waiting 30 minutes every time until previous spec files finished).
@dwelle I think you better use testFiles or --spec to select the spec files you want to run on CI, no need to remove others. Also pick the tests to run in a spec file using it.only.
@bahmutov yes, I'm doing that. But you can't use --spec in headed mode, nor does describe.only do anything when you hit the run all tests button in cypress runner. Thus debugging several specs at once in headed mode is a pain without the hacks like renaming files or temporarily removing them.
In the issue I linked, the problem turned out to be headless-only, but in order to figure that out, I had to jump through all the hoops, and this isn't the first time I had to do that.
You can use testFiles option in headed mode
Sent from my iPhone
On Oct 25, 2019, at 17:04, David Luzar [email protected] wrote:
@bahmutov yes, I'm doing that. But you can't use --spec in headed mode, nor does describe.only do anything when you hit the run all tests button in cypress runner.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
@bahmutov yea, sorry --- didn't know about that option and I don't know why I skipped over that when reading your reply. WTBS, from what it looks like the test-order will still be name dependent (but I'd have to check that) --- not that it'd matter in my case, but won't solve the issue at hand.
I want to run my most basic tests first, so that if they fail, I won't need to bother running the rest of the tests. It makes my CI fail faster and lets me start debugging earlier.
@ginna-baker you can do first "sanity" run with Cypress before running all tests, something like this
npx cypress run --spec cypress/integration/sanity-spec.js
npx cypress run
A more detailed example of the config @chloeong refers to can be found here: https://stackoverflow.com/a/59690611/2731075
