Introduce `suspenders:testing:{minitest,rspec}`

[stevepolitodesign]

Minitest

# test/test_helper.rb
ENV["RAILS_ENV"] ||= "test"
require_relative "../config/environment"
require "rails/test_help"
require "webmock/minitest"

module ActiveSupport
  class TestCase
    include ActionView::Helpers::TranslationHelper
    include ActionMailer::TestCase::ClearTestDeliveries

  end
end

WebMock.disable_net_connect!(
  allow_localhost: true,
  allow: "chromedriver.storage.googleapis.com"
)

RSpec

Generate spec/rails_helper.rb and spec/spec_helper.rb via rails g rspec:intall in an effort to not drift from what RSpec recommends out of the box.

#spec/spec_helper.rb
require "webmock/rspec"

RSpec.configure do |config|
  config.example_status_persistence_file_path = "tmp/rspec_examples.txt"
  config.order = :random

  config.expect_with :rspec do |expectations|
    expectations.include_chain_clauses_in_custom_matcher_descriptions = true
  end

  config.mock_with :rspec do |mocks|
    mocks.verify_partial_doubles = true
  end
  config.shared_context_metadata_behavior = :apply_to_host_groups
end

WebMock.disable_net_connect!(
  allow_localhost: true,
  allow: "chromedriver.storage.googleapis.com"
)

The only thing that differs from the existing spec/rails_helper.rb configuration is:

config.infer_base_class_for_anonymous_controllers = false

# spec/support/chromedriver.rb

require "selenium/webdriver"

Capybara.register_driver :chrome do |app|
  Capybara::Selenium::Driver.new(app, browser: :chrome)
end

Capybara.register_driver :headless_chrome do |app|
  options = ::Selenium::WebDriver::Chrome::Options.new
  options.headless!
  options.add_argument "--window-size=1680,1050"

  Capybara::Selenium::Driver.new app,
    browser: :chrome,
    options: options
end

Capybara.javascript_driver = :headless_chrome

RSpec.configure do |config|
  config.before(:each, type: :system) do
    driven_by :rack_test
  end

  config.before(:each, type: :system, js: true) do
    driven_by Capybara.javascript_driver
  end
end

# spec/support/shoulda_matchers.rb

Shoulda::Matchers.configure do |config|
  config.integrate do |with|
    with.test_framework :rspec
    with.library :rails
  end
end

# spec/support/i18n.rb

RSpec.configure do |config|
  config.include ActionView::Helpers::TranslationHelper
end

# spec/support/action_mailer.rb

RSpec.configure do |config|
  config.before(:each) do
    ActionMailer::Base.deliveries.clear
  end
end

Notable changes

This commit removes the formulaic dependency. A follow-up commit could explore creating a separate one-off generator for this, but for now, we're aiming for the leanest build possible.

To Do

• [ ] Add https://github.com/thoughtbot/action_dispatch-testing-integration-capybara

[stevepolitodesign]

@seanpdoyle tagging you in case you think there's anything else worth adding to test/test_helper.rb on a fresh Rails install. Anything Capybara related I might be missing?

[mike-burns]

I'd rather see a suspenders:testing generator. Why are we offering a choice between RSpec and Minitest?

[stevepolitodesign]

I'd rather see a suspenders:testing generator. Why are we offering a choice between RSpec and Minitest?

I like the idea of offering the ability to chose between test suites, since not all projects will have the same requirements. It's possible a client might have more familiarity with one suite over the other.

That being said, this could be a great opportunity to challenge our existing preferences by defaulting to Minitest over RSpec for new Rails applications. My vision for Suspenders v3.0.0 is to do as little as possible. Since Rails already ships with a completely configured test suite, we can avoid another dependency. Additionally, configuring Minitest to match the existing RSpec configuration requires less effort as demonstrated in the pull request description. For example, Rails already ships with a configured system testing framework.

Additionally, RSpec needs to play catch-up with Rails as new features are added. For example, Rails ships with testing helpers for Action Cable. When that was introduced, RSpec needed to add it, and is also burdened to keep parity with Rails.

[MatheusRich]

Worth noting rspec still doesn't support parallel testing by default.

[jho406]

I have a strong preference for RSpec. I think it has more helpful test output and a nice variety of matchers. I'm not too concerned about parallel testing as my choice of CI has parallel testing support anyway.

[mike-burns]

This would be the only Suspenders generator that offers a choice, which deviates from the goal of having Suspenders be the codification of our technical decisions.

TBH, I'm not sure what I'd use Suspenders for if it isn't making the choice for me.

[stevepolitodesign]

This would be the only Suspenders generator that offers a choice,

#1145 also introduces a choice as well. Maybe we need to clean that up too 🤷

EDIT: Although, it provides a default.

[DoodlingDev]

I agree that suspenders should be opinionated and choose one, and I (shockingly to myself of two years ago) think it should prefer Minitest. After using it on Closeknit and finding it no harder to work with than rspec (ergonomics was my main concern) I find myself enamored with ruby's built-in testing framework, no extra metaphors, just ruby code.

Also feels easier to back out of if one truly wants rspec.

[emilford]

I don't have nearly as much experience using Minitest, but I'm in the camp of having it as our default—eventually. I'm a fan of one less dependency and a test framework that doesn't need to play catch up, amongst other pros.

Eventually because most of the team is far more comfortable with RSpec. The needs here may be small, but I'm curious of the other side of this conversation: what we may need to do to smooth a transition to Minitest.

Part of having suspenders—to @mike-burns's point—is to remove need to make decisions, but also for consistency across projects. Would having both options further the divide of projects using different testing tools?

[thiagoa]

Minitest is great, but I don't think we should use it by default. The reason is that it's too barebones and requires helper gems to come a bit closer to what RSpec provides by default, such as advanced matchers, run only failed tests, full-fledged test runner with documentation, etc. Every time I use minitest on a reasonable project, I add more and more gems to cover my needs. I actually love Minitest, but it has these gotchas!

Also, thoughtbot is already familiar with RSpec. There is a lot of learning material in Upcase, our blog, etc.

[jho406]

Additionally, RSpec needs to play catch-up with Rails as new features are added.

I'd like to think that MiniTest needs to play catch up with RSpec 😄 . Stubbing, mocking, extremely convenient matchers, etc.

Generally, i think there has to be a really compelling reason to switch testing tools. If it helps create better tests with less common gotchas (less mystery guests, instance variables, mixins) I would be game, but if its just that its part of Rails stack or that its in ruby, that's not enough.

[emilford]

Generally, i think there has to be a really compelling reason to switch testing tools. If it helps create better tests with less common gotchas (less mystery guests, instance variables, mixins) I would be game, but if its just that its part of Rails stack or that its in ruby, that's not enough. If it helps create better tests with less common gotchas...

+1 on a compelling reason. Writing more maintainable tests has historically been an argument in favor of Minitest. May have changed over the years, but I also remember preferring/missing mocking in RSpec.

@thiagoa what gems do you find yourself adding on every project? They could be a part of the defaults. I must have my --next-failure and --only-failures equivalent. 😆 I use both regularly.

[seanpdoyle]

if its just that its part of Rails stack or that its in ruby, that's not enough.

In my opinion, this is the most compelling reason. https://github.com/thoughtbot/suspenders/pull/1156#issuecomment-1912168496 shares a great example.

To share a concrete example, 3 years ago, Rails recently introduced the concept of an Error Reporting object as part of the 7.0 release. Alongside that concept, it also added some built-in assertions as part of ActiveSupport::Testing::ErrorReporterAssertions. Does RSpec Rails have a similar set of matchers?

Out of the box, Rails provides built-in general purpose Minitest extensions as part of ActiveSupport::Testing::Assertions, and other utilities like ActiveSupport::Testing::TimeHelpers and ActiveSupport::LogSubscriber::TestHelper.

Along with those universal utilities, there are framework specific test cases, like:

• ActionDispatch::SystemTestCase
• ActionDispatch::IntegrationTest
• ActionMailer::TestCase
• ActiveJob::TestCase
• ActionCable::Connection::TestCase and ActionCable::Channel::TestCase
• ActionView::TestCase

Along with those test cases, there are framework specific helpers to mix-and-match, like:

• https://edgeapi.rubyonrails.org/classes/ActionMailer/TestHelper.html
• https://edgeapi.rubyonrails.org/classes/ActionMailbox/TestHelper.html
• https://edgeapi.rubyonrails.org/classes/ActionCable/TestHelper.html
• https://edgeapi.rubyonrails.org/classes/ActiveJob/TestHelper.html

Since these are all part of the framework, they don't require additional layers of abstraction or other integration touchpoint. Most rspec-rails versions of these concepts (like type: :system or type: :request) end up relying on the internals of the test cases (for example, type: :request, type: :system, and type: :view).

In my mind, consuming the helpers directly from Rails means teams are more encouraged to read the Rails Guides and API documentation. If we find the tooling to be poorly documented, or behaving in unexpected ways, making those improvements upstream will benefit a larger group of end-users than making improvements to a project like RSpec that's a downstream dependency. (I also strongly encourage everyone to make documentation and implementation improvements to RSpec too!).

Learning and using Minitest had the largest impact on my abilities to make contributions to rails/rails, since the framework itself utilizes minitest, and making those contributions helped familiarize myself with the built-in helpers that are meant for public consumption. It has truly become a virtuous cycle that I'd encourage others to try and experience.

[thiagoa]

@emilford I don't remember, as I don't regularly work with Minitest projects. But at the very minimum, a better test runner and better matchers for more powerful and expressive assertions (maybe that changed?). I'm pretty sure error reporting is better on RSpec as well. I mostly used Minitest in small-scale personal projects, otherwise, I always pulled additional gems.

[mike-burns]

To echo what others are saying, if we think Minitest is better but it requires more setup and gems installed, that's exactly what we should be doing in Suspenders.

[thiagoa]

To be more specific, here are some features I would miss in Minitest:

• match_, have_, etc, assertions. In Minitest, it's common to do something like assert_equal expected_array, actual_array.sort instead of expect(actual_array).to match_array(expected_array). RSpec is more expressive and provides a great diff, while Minitest does not;
• aggregate_failures - Very powerful and provides excellent error reporting. It also allows for a more optimized style of writing tests where one asserts the complete behavior of a method in a single test while not compromising on error reporting;
• Mocking and stubbing - rspec-mocks is better than Minitest::Mock. The latter might be elegant, yes, but it's far from being full-featured;
• Run test by line - This is basic, and I suspect Rails' test runner covers it. Minitest, by default, does not, as far as I'm concerned;
• Run only failed tests - Not sure if Rails test runner provides that;
• In Minitest, I usually write more helpers for basic things that should already be covered;
• RSpec's diffing and error reporting is fantastic, especially when using more powerful matchers. Minitest error reporting is basic and not great.

I'm sure there are more. These are all areas where I would resort to additional gems, if available.

On the other hand, I love that Minitest does not have let, before, and other problematic RSpec constructs (not talking about minitest-spec), which would force us to avoid them. My feeling is that Minitest is great for more technical projects like Rails, but for consulting and client projects not so.

[stevepolitodesign]

@thiagoa

To be more specific, here are some features I would miss in Minitest:

Like @mike-burns said, but I wonder if this would provide an opportunity to add those features now as part of this generator?

[thiagoa]

Just to be extra clear, I prefer xUnit-like tests to spec-style tests. I'm not a big fan of expect and that verbose style that tries to resemble English. The main point is that RSpec is more full-featured and provides a better testing experience.

[thiagoa]

@stevepolitodesign If we can find something that covers the more urgent gaps, such as a decent test runner, better diffing and error reporting, more powerful assertions, etc, it may be a good opportunity! I can certainly live without aggregate_failures even though it's great to have it. I need to catch up with the Minitest ecosystem.

[jho406]

Another scenario to think about. If we default to mini-test as the preferred test suite and ADD gems to make up for https://github.com/thoughtbot/suspenders/pull/1156#issuecomment-1912622007. Then all were doing is removing rspec, a robust active project, and adding a bunch of smaller gems that may not see the equivalent maintenance. 1 less dependency, but X more.

So my question quite similar to this comment https://github.com/thoughtbot/suspenders/pull/1156#issuecomment-1912553705. If we do go with minitest, what smaller are gems are we adding and how is it maintained?

[purinkle]

👀 I have always seen Suspenders as thoughtbot's opinionated Rails configuration.

I agree with @mike-burns that we should choose Minitest or RSpec and back that decision.

It would interest me to hear where this groundswell of Minitest love originated.

I've heard nothing from the developers I work with.

Is there an echo chamber of opinion?

I have no strong preference, but I would love more context. 🤔

[stevepolitodesign]

It sounds like there are two issues we need to address:

1. Suspenders should set a default test framework for new Rails applications.
2. We are divided on what test framework to default to.

With that in mind, it let me pose another question: Would people feel more comfortable if we allow the choice between RSpec and Minitest, but default to RSpec?

[mike-burns]

allow the choice

To me, the benefit of Suspenders is that I don't choose things.

[stevepolitodesign]

@thiagoa I would love to explore some solutions to your concerns in a way that would still reduce the number of dependencies.

In Minitest, I usually write more helpers for basic things that should already be covered;

I wonder if we could include some generic helpers with this generator?

[mike-burns]

Back to the Minitest vs Rspec question: how many Minitest projects have we worked on recently? Who is writing Minitest on new projects?

[MatheusRich]

My stats: I have never worked with a client using Minitest.

I've seen some usage on gems.

[MatheusRich]

According to the 2022 Rails community survey (from Planet Argon), RSpec is used around 3x more than Minitest.

[stevepolitodesign]

@purinkle

It would interest me to hear where this groundswell of Minitest love originated.

I can only speak for myself, but we're having good luck with Minitest on a long-running client project that's in production.

Is there an echo chamber of opinion?

Possibly? Since we currently seem to prefer RSpec as an organization, I'm guessing there aren't many opportunities to use Minitest.

Just noting that this very re-write of Suspenders is using Minitest because that's what ships with a Rails Engine. This isn't our only open-source project that is doing the same. https://github.com/thoughtbot/capybara_accessibility_audit is another example.

[jho406]

Alongside that concept, it also added some built-in assertions as part of ActiveSupport::Testing::ErrorReporterAssertions. Does rspec Rails have a similar set of matchers?

No, but If we pick things that mini-test has that rspec doesn't has, would it be fruitful to list out things rspec has that minitest doesn't have? I'm not sure who needs to catch up to who.

As @thiagoa said, and @emilford alluded to I think the lack of lets and lets! gives mini-test an edge. And i agree with that point, but i'm also left wondering if the same mistakes one can make in RSpec can also be made in minitest.

On top of what @MatheusRich said about RSpec being 3x more prevalent, and the need to supplement minitest with other gems to reach parity with our dev workflow. Its hard to see the strong compelling reason.

[stevepolitodesign]

After reviewing feedback in this pull request, I decided to keep RSpec over the default Rails test framework for the following reasons:

• This is the exiting behavior for Suspenders
• We have a historical precedent for using RSpec, and team members may not be as familiar with Minitest.

However, since Suspenders is a living, breathing representation of thoughtbot's opinions, we can continue to explore if and how we would want to use the default Rails test framework by making necessary modifications to meet our needs.