Rails Capybara browser tests broken by previous test states

[jackc]

My test suite has been plagued by flickering failures for the last month or two. Failing tests would always succeed when run alone. Previous versions of the application that were known to pass their tests no longer did. Presumably, this means there was a change in Chrome or the underlying OS. So I'm not sure if this is actually a Capybara, Chrome, or Rails issue, but my guess is even if it is an upstream problem it may need to be worked around here.

The failure only occurs in headless Chrome. Visible Chrome worked.

assert_text would fail but the error message would state that the text was found in non-visible text. The error screenshots showed the "non-visible" text. Using the debugger I was able to see that the "non-visible" text was available when the error message was being built. This seemed to indicate some kind of race condition. However, timing changes did not affect the results. This led me to guess something in the browser, driver, or Capybara state was being corrupted.

After extensive testing and debugging I was able to reduce and reproduce the test failures to combinations of 2 tests. Two things would cause failures:

1. Clicking a link with target="_blank" (i.e. opening a new tab)
2. Visiting a page with onbeforeunload

The onbeforeunload cause is not too surprising as it might leave a blocking modal dialog in the way. But target="_blank" was very surprising. I tried to close the new browser tab but it made no difference. And when inspecting windows only showed one window in the 2nd test that actually failed. But it seems that somehow Capybara / Selenium / webdriver / Chrome is getting confused. visit / fill_in / click_on all work properly and then assert_text is somehow failing due to there having been another tab open.

I was able to work around the issue in my test suite by adding the following code to the end of each test that used the offending functionality:

    ensure
      page.quit

Again, I'm not sure exactly where this behavior should be remedied or documented or exactly what's happening at the lower levels, but this was an issue that took me about a day to track down. Hopefully, this issue at least saves someone else some time.