Question: how do you feel about Selenium for integration tests?

[aickin]

So, I have a question for you all: how would you feel about porting the current integration test codebase from Zombie to Selenium?

I ask because I have a few features I'm itching to add, but I find myself a bit worried about how faithfully Zombie will test various scenarios. This is especially true with feature ideas like using the fetch API for data or implementing http/2, when different browsers end up taking completely different code paths because of polyfills and differing capabilities. I've been nervous enough about the testing matrix that I feel like it's worth stopping and shoring up the integration suite before proceeding. The only way I can think of doing so that makes sense to me is through Selenium and (probably) SauceLabs.

(It's worth noting that I wrote the first version of the integration suite, and I specifically opted for Zombie and against Selenium because I'd had bad experiences with maintaining Selenium tests and test machines in the past. I also was putting more weight on simplicity and reproducibility of tests back then and less on cross-browser quirks.)

Selenium pros:

• Runs on real browsers: As we get into more optimizations and features, it's more and more likely that we hit browser inconsistencies. For example, if we move ReactServerAgent over to the Fetch API, it'd be nice to have integration tests that run both in browsers that natively support Fetch and ones that use the polyfill.
• Takes videos and screenshots by default: Every test that runs on sauce labs takes a video of the browser as it goes through the test. This can make it much, much easier to understand what failed in a test.
• Our tests seem convertible: I've looked through our current integration test codebase, and it seems like all or almost all of the current tests would be able to be converted to Selenium.
• Setting up Travis seems relatively easy: Setting up with Selenium/Sauce/Sauce Connect is pretty easily supported on Travis.

Selenium cons:

• A lot slower: Each command to the browser is run independently, over the Internet, as opposed to in-process JavaScript with Zombie. It's fairly easy to parallelize different testing different browsers in Travis, but parallelizing individual tests is harder.
• More confusing to run tests locally: Zombie just requires npm install && gulp test locally, whereas Selenium would require either a webdriver/selenium server local installation, or that the dev have Sauce Lab credentials and set up Sauce Connect.
• More moving parts makes tests less reliable: I have a general worry that the more complexity you put into your test setup, the more likely it will fail intermittently, and intermittent failures are the death of a test codebase.

[roblg]

I'm not opposed to investigating a switchover, for sure. I love the ease of use of zombie, but I definitely don't feel confident that it's going to catch anything browser-specific.

re: cons -

A lot slower: Each command to the browser is run independently, over the Internet, as opposed to in-process JavaScript with Zombie. It's fairly easy to parallelize different testing different browsers in Travis, but parallelizing individual tests is harder.

I'd be interested to get an idea of how much slower they are. I know they're slower than running in zombie, but our tests aren't doing that much. Hopefully it wouldn't be awful?

More confusing to run tests locally: ... Selenium would require either a webdriver/selenium server local installation, or that the dev have Sauce Lab credentials and set up Sauce Connect.

How much more confusing? I haven't ever had to set up selenium tests locally, but internally we have Java-based selenium tests that are run just like JUnit tests; basically no additional configuration needed. I would hope that we could get something working like that in this case? I think not being able to easily run tests locally would probably be a deal-breaker to getting people to write new ones.

[aickin]

To make this discussion more concrete, as an experiment I converted the (very simple) HelloWorld test spec to Selenium/Sauce; you can see it here. A few thoughts:

• The tests look pretty good: I think the actual test code in the spec looks pretty good/readable, especially considering it all became much more async. Might be even better if we integrated jasmine-promises, which allows you to just return a Promise from a test rather than mucking around with done().
• Write once, test on many renders works: I really liked the strategy in the existing tests of writing a test once that can be run three times, once each on the server render, client render, and page-to-page transition. That strategy is still doable on Selenium/Sauce.
• Added complexity has a cost: As with all projects like this, there were integration oddities that took a while to figure out; I think most of the complexity is hidden now from the test writer, but it could definitely rear its head again.
• OSX tests are pokey, but mobile tests are SLOOOOOOOOOOOOOOOOOW: I've observed a huge perf difference depending on the OS/browser combo (rough times per test in the HelloWorldSpec):
  • Linux/Chrome: 3-6s
  • Win 10/Chrome: 5-9s
  • Linux/Firefox: 8-11s
  • Win 10/IE 11: 8-12s
  • OSX/Safari: 12-15s
  • OSX/Chrome: 12-15s
  • Win 10/Edge: 10-12s (with a couple test failures)
  • Android/Android Browser: 53-67s
  • iOS/Safari: 58-65s

Those mobile numbers are eye-popping; it would take hours to run even a reasonably modest test suite on mobile browsers. According to this thread, the startup times aren't unusual, and it's that the simulator frequently boots slowly and crashes. There's a potential that you could start up just one simulator for a whole Spec or Suite, although that obviously runs a risk of introducing test dependencies.

[aickin]

I'd be interested to get an idea of how much slower they are. I know they're slower than running in zombie, but our tests aren't doing that much.

Was working on that last post for like the last 2 hours; collided in midair with your question.

How much more confusing?

Haven't tried it yet, but I think it involves installing the Selenium Server jar, the Chrome WebDriver, and then starting up the jar with Java before running the tests. Or something like that.

[aickin]

Actually, I just found webdriver-manager, which claims to be an npm installable package that handles all you need to get a local Selenium Server going. It's forked out of protractor, which seems to be a well-used project.

[doug-wade]

I don't think this needs to be an either/or question. I'd like to see a setup where during local development we run the integration tests against a headless browser, and on the CI server we run against Selenium in real browsers. This allows local development to continue to be as fast or faster, while improving our safety on PRs.

[aickin]

@doug-wade Thanks for the response! I want to make sure I understand what you're saying, though. Do you mean:

1. that there should be a disjoint set of tests A & B, where A tests run on Selenium and only run in CI and B tests don't use Selenium at all and run both locally and in CI? or
2. that there should be a single set of tests that all use Selenium, but locally we run them against a headless browser and in CI we run them against a browser matrix?

And a follow-on question if you meant 2: do you have a suggestion for a headless browser? AFAIK Zombie doesn't support Selenium, which is why I was thinking of it as an either-or kind of thing. Thanks!

[doug-wade]

iiuc, selenium automates any browser that has a driver, headless or otherwise, which means we could run our selenium tests on phantomjs locally, and on real browsers for ci. I think the Phantom driver would work okay, though we could try to HtmlUnit driver or EventFiringWebDriver, which both seem headless.

At second reading, my first response was needlessly confusing. What I meant was: I think moving to Selenium is a good idea because we could still use headless browsers to do local testing, which should be relatively quick.

I also believe that we can make setup pretty simple, using webdriver-manager and phantom-prebuilt and selenium-webdriver, such that you wouldn't have to leave npm and gulp to do your testing/setup.

[aickin]

Thanks, @doug-wade, that clarifies things for me, and I think we're basically on the same page.

One question for you: how do you feel about testing with a non-headless Firefox? I've been experimenting with webdriver-manager, and I'm pretty impressed at how well it seems to work with Firefox and Chrome, and Travis supports running Firefox locally with xvfb. I even just opened an experimental PR that runs some of the existing tests against FF/xvfb on Travis, and the build passed (much to my surprise).

It also seems to work locally, although it may require Firefox to be on the dev box. Thoughts on that?

[gigabo]

Looks like #102 is related to this.

[aickin]

Indeed. It was (afaict) totally working, but then it started to be a little flaky, so I didn't think it should merge yet. Then since last Wednesday, I've been sidetracked by my PR for the first step of integrating streaming into React core.

[gigabo]

PR for the first step of integrating streaming into React core.

✨ Awesome! ✨

/me furiously clicks emoji on that PR.