feat: experimental single tab run mode for component testing

[lmiller1990]

• Resolves https://github.com/cypress-io/cypress/issues/22353

User facing changelog

Introduce a new Cypress experiment, singleTabRunMode, for Component Testing. All specs will execute in a single tab, similar to the Component Testing alpha in Cypress 9.

Additional details

Useful Documentation

• Technical brief (Cypress team only)
• Documentation Update

In this PR, I added an experimental flag that will reverted back to the Cypress 9 behavior, which will run all the specs for CT in a single tab. It's much faster (see the associated issue for numbers). There are some trade-offs, as noted in the technical brief, which is why this is behind an experimental flag.

This experiment was driven by feedback in https://github.com/cypress-io/cypress/issues/22353, and also from our dashboard users, noting running all CT specs was up to 50% slower.

Ideally, when we work on the Run All specs feature, we will explore and find out how to best isolate specs running in a single tab. This work will likely make this experiment obsolete, and we'd just deprecate it, in favor of the new run all specs feature. This is a significant project and unlikely to land anytime soon, thus this experiment.

Steps to test

• pull this branch down
• go in packages/launchpad, which enables the experiment
• yarn cypress:run:ct --browser chrome --config video=false. 55s on my machine. :zap:
• observe it's faster, and it runs all specs in 1 tab. You can add --headed to see it running in a single tab.
• remove experimental flag from cypress.config.ts
• yarn cypress:run:ct --browser chrome --config video=false. 91.9s on my machine. :snail:

How has the user experience changed?

Component Tests run much more quickly with this experiment enabled.

PR Tasks

• [x] Have tests been added/updated?
• [x] Has the original issue (or this PR, if no issue exists) been tagged with a release in ZenHub? (user-facing changes only)
• [x] Has a PR for user-facing changes been opened in cypress-documentation?
• [x] Have API changes been updated in the type definitions?

[cypress-bot[bot]]

Thanks for taking the time to open a PR!

• Create a Draft Pull Request if your PR is not ready for review. Mark the PR as Ready for Review when you're ready for a Cypress team member to review the PR.
• Become familiar with the Code Review Checklist for guidelines on coding standards and what needs to be done before a PR can be merged.

[cypress[bot]]

---

Test summary

37828 • 0 • 615 • 0 • 7

---

Run details

Project	cypress
Status	Passed
Commit	56bf522dbf
Started	Aug 16, 2022 12:05 AM
Ended	Aug 16, 2022 12:25 AM
Duration	20:24 💡
OS	Linux Debian - 11.3
Browser	Multiple

View run in Cypress Dashboard ➡️

---

Flakiness

	cypress/e2e/commands/net_stubbing.cy.ts		5
1	network stubbing > intercepting request > can delay with deprecated delayMs param
2	network stubbing > intercepting request > can delay with deprecated delayMs param
3	network stubbing > intercepting request > can delay with deprecated delayMs param
4	network stubbing > intercepting request > can delay with deprecated delayMs param
5	network stubbing > intercepting request > can delay with deprecated delayMs param
This comment includes only the first 5 flaky tests. See all 7 flaky tests in the Cypress Dashboard.

---

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

[lmiller1990]

@emilyrohrbough thanks for the review. @brian-mann said he's going to propose an alternative to what I'm doing in this PR, so I'll wait until that comes before going too deep into any of your recommended changes.

[astone123]

Running the CT specs in launchpad with this experiment went from ~46s to ~28s on my machine 👍🏻

[lmiller1990]

@ryanthemanuel thanks for the quick review! Updated.

Did some manual QA on the video recording, seems to be working as expected.

[cypress-bot[bot]]

Released in 10.6.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to Cypress v10.6.0, please open a new issue.

[brian-mann]

@lmiller1990 can you upload a screenshot of what the settings UI looks like for E2E vs CT that indicate which experiments are available?

I would expect that experimentalSingleTabRunMode experiment option is displayed as being available to set in CT, but not in E2E. Additionally because this is a testingType specific option, I would expect the docs to explain this and the option to guide the user towards setting it in the right place (not the top level).

Currently the UI looks like this:

Notice that there isn't a place where it explains where to set the experimental flags. Since we now are building support for testingType specific options I would expect the UI to accommodate this.