Re-run failures only

[gap777]

Is it possible to re-run the tests limited to those that failed on the last execution?

[ilyazub]

@gap777 I use rspec --only-failures for that. Currently, this is not supported with turbo_tests.

I expect that passing the extra options to RSpec will be enough to support this feature (https://github.com/serpapi/turbo_tests/issues/14#issuecomment-898405912).

[douglasshuang]

I work with @gap777.

I realized that rspec --only-failures couldn't find failed examples reliably because the parallel runners invoked by turbo_tests were writing to the same status file and clobbering each other's output. I changed our spec/spec_helper.rb to use a different file name per runner:

  config.example_status_persistence_file_path = "spec/examples#{ENV['TEST_ENV_NUMBER']}.txt"

We already use a shell script to invoke turbo_tests, so I added a liine to consolidate those files at the end:

cat spec/examples?*.txt > spec/examples.txt

So for now, we can at least run (non-parallel) rspec --only-failures after turbo_tests.

[ilyazub]

@douglasshuang Thanks to your feedback!

Please note that combining multiple examples*.txt may lead to the RSpec failure. RSpec status persistence files have a header which is expected to be present once (ref: ExampleStatusParser#initialize).

# Concatenate multiple files with the same header
# https://unix.stackexchange.com/a/170692/33308
$ (head -2 spec/examples1.txt && tail -n +3 -q spec/examples?*.txt) > spec/examples.txt

# Checking it worked
$ wc -l spec/examples*.txt
   73015 spec/examples.txt
   46467 spec/examples1.txt
   26550 spec/examples2.txt
  146032 total

# Header is present only once
$ grep example_id spec/examples*.txt
spec/examples.txt:example_id                                                                                                 | status  | run_time               |
spec/examples1.txt:example_id                                                                                                 | status  | run_time               |
spec/examples2.txt:example_id                                                                                                 | status  | run_time               |

[douglasshuang]

@ilyazub Thank you very much for the refinement!