Audit failures should not stop the test from running

[pangolingo]

I've observed that if a test fails because a page is not accessible, the remainder of the test block does not run. This results in issues being reported for a single page, but not for any other pages loaded in a test block.

Expected behavior: The test should run to the end (barring other non-accessibility failures), even if a few of the pages visited fail the accessibility audit. Audit errors should be reported across all pages visited.

Actual behavior: The test halts on the first accessibility error that it finds

Example:

RSpec.describe "Playbook", type: :system do
  around do |example|
    with_accessibility_audit_options do
      example.run
    end
  end

  it "renders all pages without error" do
    visit "/playbook"

    playbook_links = all(".application-main-content a[href^='/playbook']").map do |link|
      link[:href]
    end
    playbook_links.each do |url|
     # the first page visited that has an accessibility error will end the test
      visit url
    end
  end
end

I do realize that this issue/suggestion goes against the default behavior of Capybara - Capybara normally does not run the entire test if an assertion fails, or if it fails to visit a page. But it would be a great enhancement.