Support for using Queue mode with minitest/spec

[caspark]

We use minitest/spec from https://github.com/seattlerb/minitest together with minitest, and that breaks Queue mode, because context, before, etc aren't defined:

D, [2019-11-15T14:44:55.150743 #9554] DEBUG -- : [knapsack_pro] API request UUID: 9166cd89-c25f-4c42-b103-abb3b2d38822
D, [2019-11-15T14:44:55.150854 #9554] DEBUG -- : [knapsack_pro] API response:
D, [2019-11-15T14:44:55.150921 #9554] DEBUG -- : [knapsack_pro] {"queue_name"=>"987:3924a8d106184efbc955b3890d762fd1", "build_subset_id"=>nil, "test_files"=>[{"path"=>"test/system/subscriber_notifications/subscriber_notifier_test.rb", "time_execution"=>nil}]}
trying to run ["test/system/subscriber_notifications/subscriber_notifier_test.rb"]
D, [2019-11-15T14:45:01.478340 #9554] DEBUG -- : [knapsack_pro] Test suite time execution queue recording enabled.
rake aborted!
NoMethodError: undefined method `context' for SubscriberNotifierTest:Class
Did you mean?  const_set
/home/caspark/src/status-page-web/test/system/subscriber_notifications/subscriber_notifier_test.rb:8:in `<class:SubscriberNotifierTest>'
/home/caspark/src/status-page-web/test/system/subscriber_notifications/subscriber_notifier_test.rb:5:in `<main>'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `block in require'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in `load_dependency'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/knapsack_pro-1.15.0/lib/knapsack_pro/runners/queue/minitest_runner.rb:89:in `block in minitest_run'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/knapsack_pro-1.15.0/lib/knapsack_pro/runners/queue/minitest_runner.rb:88:in `each'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/knapsack_pro-1.15.0/lib/knapsack_pro/runners/queue/minitest_runner.rb:88:in `minitest_run'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/knapsack_pro-1.15.0/lib/knapsack_pro/runners/queue/minitest_runner.rb:67:in `run_tests'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/knapsack_pro-1.15.0/lib/knapsack_pro/runners/queue/minitest_runner.rb:30:in `run'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/knapsack_pro-1.15.0/lib/tasks/queue/minitest.rake:6:in `block (3 levels) in <main>'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/cli/exec.rb:74:in `load'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/cli/exec.rb:74:in `kernel_load'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/cli/exec.rb:28:in `run'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/cli.rb:465:in `exec'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/cli.rb:27:in `dispatch'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/cli.rb:18:in `start'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/exe/bundle:30:in `block in <top (required)>'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/exe/bundle:22:in `<top (required)>'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/bin/bundle:23:in `load'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/bin/bundle:23:in `<main>'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/bin/ruby_executable_hooks:24:in `eval'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/bin/ruby_executable_hooks:24:in `<main>'
Tasks: TOP => knapsack_pro:queue:minitest
(See full trace by running task with --trace)

We do require 'minitest/spec' as one of the first things in our test_helper.

The regular Knapsack Pro runner works fine, but it looks like the Knapsack Pro Queue runner reimplements minitest's test running.

[ArturT]

@caspark What is your version of minitest gem?

[caspark]

Sorry, should've included that already. Here are all the minitest-related gems we're using:

guard-minitest-2.4.6
minispec-metadata-3.3.1
minitest-5.13.0
minitest-reporters-1.4.2
minitest-retry-0.1.9
minitest-spec-rails-5.4.0
minitest-stub_any_instance-1.0.1

[ArturT]

Can you double-check you have RAILS_ENV=test set before running queue mode?

export RAILS_ENV=test

[ArturT]

Here is how I run tests in queue mode for minitest https://github.com/KnapsackPro/knapsack_pro-ruby/blob/6849d1fd2610d2d7469ae2a7148ab8ef5faab4c7/lib/knapsack_pro/runners/queue/minitest_runner.rb#L56

Maybe I should require 'minitest/spec' as well https://github.com/KnapsackPro/knapsack_pro-ruby/blob/6849d1fd2610d2d7469ae2a7148ab8ef5faab4c7/lib/knapsack_pro/runners/queue/minitest_runner.rb#L6 but not sure if I should do it always even when someone is not using minitest/spec.

I'll try to play more with it tomorrow morning.

[caspark]

Yep, definitely setting RAILS_ENV - invocation is along the lines of env CI_NODE_TOTAL=2 CI_NODE_INDEX=0 RACK_ENV=test KNAPSACK_GENERATE_REPORT=true KNAPSACK_REPORT_PATH=$KNAPSACK_REPORT_PATH KNAPSACK_PRO_BRANCH=SIP-799-knapsack-pro-hacking-local KNAPSACK_PRO_COMMIT_HASH=$(git rev-parse HEAD) KNAPSACK_PRO_TEST_SUITE_TOKEN_MINITEST=<replaced> bundle exec rake knapsack_pro:queue:minitest.

[ArturT]

I managed to reproduce this bug on my local repo.

Can you update your test/test_helper.rb and add there

 # Allow context to be used like describe
  class << self
    alias :context :describe
  end

I found this tip https://stackoverflow.com/a/25811713/905697

This is from my test_helper.rb:

class ActiveSupport::TestCase
  # Setup all fixtures in test/fixtures/*.yml for all tests in alphabetical order.
  fixtures :all

  # Add more helper methods to be used by all tests here...
  extend MiniTest::Spec::DSL

  # Allow context to be used like describe
  class << self
    alias :context :describe
  end

  register_spec_type(self) do |desc|
    desc < ActiveRecord::Base if desc.is_a?(Class)
  end
end

https://github.com/KnapsackPro/rails-app-with-knapsack_pro/blob/8f8d55a636f3b59d9558be8f0bc39e6bdb25c817/test/test_helper.rb#L30,L47

[ArturT]

Also, I found out if you use RSpec and minitest at the same time then RSpec overrides some stuff and minitest test inside of describe blocks are not executed at all.

https://stackoverflow.com/questions/23683009/minitest-not-picking-up-describe-blocks

I had to disable loading RSpec when running minitest with ENV: https://github.com/KnapsackPro/rails-app-with-knapsack_pro/blob/8f8d55a636f3b59d9558be8f0bc39e6bdb25c817/Gemfile#L65,L69

This is my example minitest test file with spec syntax describe/context https://github.com/KnapsackPro/rails-app-with-knapsack_pro/blob/8f8d55a636f3b59d9558be8f0bc39e6bdb25c817/test/minitest/meme_spec_test.rb#L12,L17

[ArturT]

@caspark Did my tips help?

[ArturT]

@caspark Can you try the solution I posted? Thank you.

[caspark]

Sorry @ArturT , have been busy with other things. I suspect that aliasing context -> describe won't be sufficient (context being undefined happened for this test case, but something else was undefined for another test case), but haven't tried it yet. Unfortunately with some PTO coming up I likely won't have time to try it any time soon. You can close this issue in the meantime, and if this approach doesn't work then I'll comment again when I have time to investigate further.

[ArturT]

@caspark I will keep this issue open. You can get back to it when you find some time.

Do you use another test runner like RSpec? I noticed that loading RSpec gem breaks minitest. See my previous tip.

[shadre]

@caspark have you had a chance to revisit this issue?

[caspark]

No, sorry, haven't looked at this since.

(Also, no, not using an alternative test runner like rspec at the moment.)

[ArturT]

@caspark If there is anything we can help with, please let me know. If you would revisit this issue that would be helpful. Otherwise, we are going to close this issue.