turbo_tests icon indicating copy to clipboard operation
turbo_tests copied to clipboard

Add option to print failing group

Open mRudzki opened this issue 1 year ago • 1 comments

Why this option is useful? If you stumble across random failures you can debug it easier.

mRudzki avatar May 12 '23 07:05 mRudzki

I did it separately from https://github.com/serpapi/turbo_tests/pull/33 but maybe I shouldn't . Many projects I worked on have some old/poorly written specs that fail only on specific occasion.

Example 1

it "one" do
  record = build :record
  expect { record.save }.to eq(true)
end

it "two" do
  record = build :record, id: 1
  expect { record.save }.to eq(true)
end

In this simple example, order "two, one" will succeed but "one, two" will fail.

Example 2

A more real world (I had that exact problem on some old spec last month) example is setting thread variables and forgetting to unset them later.

# file1.rb
let(:client) { create :client }
...
it "saves currrent_user_id in execution_context_store" do
  expect { complex_class.new(user: client).call }.to change(Thread.current[:execution_context_store]).from(nil).to({client: client})
end

# file2.rb
  after do
    Thread.current[:execution_context_store] = {}
  end

  describe ".store" do
    before do
      Thread.current[:execution_context_store] = { some: "value" }
    end

    it "returns value stored in current thread 'execution_context_store'" do
      expect(described_class.store).to eq(some: "value")
    end
  end

If file1.rb will be on same thread will be before file2.rb and won't be cleared with "after" we will have an error.

Why would it be beneficial?

Printing failing group, prints tests that were in that group. If we combine it with seed option, we can do bundle exec rspec --bisect and get the failing and specs it is depending on. Without it we have to run the whole suite, and even if we'll know failing seed on turbo_tests we won't be able to bisect on it. Here is why:

.
├── spec
│   ├── services
│   │   ├── file1.rb
│   │   ├── file2.rb
│   │   ├── file3.rb
│   │   ├── file4.rb
└── ...

Let's assume we have an example 2 but we also have:

# file3.rb
  after do
    Thread.current[:execution_context_store] = {}
  end
 ...

Let's assume turbo_tests seed 4321 -n 2 will return order: Thread1: "file1.rb, file2.rb", Thread2: "file3.rb, file4.rb" Result fail Let's assume "normal" rspec seed 4321 will return order: "file1.rb, file3.rb, file2.rb, file4.rb" Result: no fail because file3.rb prevented failure

mRudzki avatar May 29 '23 09:05 mRudzki