Seed value is no longer printed starting in version 2.2.4

[aprescott]

Version 2.2.4 appears to no longer display any seed information output, for success or failure, either at the beginning or the end of the run:

Finished in 7.38 seconds (files took 3.17 seconds to load)
467 examples, 0 failures

Finished in 5.24 seconds (files took 1.98 seconds to load)
467 examples, 1 failure

Failed examples:

rspec ./spec/models/foo_spec.rb:56 # 1 is expected to eq 2

Compare that to 2.2.3, which outputs at the start and the end of the test, as expected:

1 processes for 1 specs, ~ 1 specs per process

Randomized with seed 21152

# ...

Failures:

  1) Foo is expected to eq 2
     Failure/Error: specify { expect(1).to eq(2) }
     
       expected: 2
            got: 1
     
       (compared using ==)
     # ...

Finished in 5.47 seconds (files took 2 seconds to load)
467 examples, 1 failure

Failed examples:

rspec ./spec/models/foo_spec.rb:56 # Foo is expected to eq 2


Randomized with seed 21152

1 processes for 1 specs, ~ 1 specs per process

Randomized with seed 37227

# ...

Finished in 5.7 seconds (files took 2.06 seconds to load)
467 examples, 0 failures


Randomized with seed 37227

[aprescott]

Since 2.2.4 includes #48 as the only documented change, I assume something there broke.

[aprescott]

Meant to include other version information, apologies! 😅

$ bundle exec rspec --version
RSpec 3.13
  - rspec-core 3.13.0
  - rspec-expectations 3.13.1
  - rspec-mocks 3.13.1
  - rspec-rails 6.1.3
  - rspec-support 3.13.1

$ ruby -v
ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) [arm64-darwin23]

Happy to provide any other extra information if it's helpful.

[ilyazub]

Please add the --seed NUMBER parameter to the turbo_tests command. Before #48, seed was randomized by default, which was confusing. (ParallelTests and RSpec apply seed only when option is used explicitly.)

Here's the output with the --seed parameter

$ bundle exec turbo_tests --seed $RANDOM
Using recorded test runtime
3 processes for 3 specs, ~ 1 specs per process

Randomized with seed 5366
..........

Finished in 4.56 seconds (files took 3.12 seconds to load)
10 examples, 0 failures

Randomized with seed 5366

Output without the seed parameter

$ bundle exec turbo_tests 
Using recorded test runtime
3 processes for 3 specs, ~ 1 specs per process
..........

Finished in 4.74 seconds (files took 3.11 seconds to load)
10 examples, 0 failures

Thanks for the bug report @aprescott and sorry for the inconvenience.

Please let me know if adding the --seed NUMBER parameter works for you.

[aprescott]

RSpec apply seed only when option is used explicitly.

This option is enabled for rspec runs through the standard spec_helper.rb config boilerplate:

config.order = :random
Kernel.srand config.seed

Moreover, turbo_tests is using this config: the ordering of tests is still random when using turbo_tests, it's just that now there's no seed value shown. If I use version 2.2.4 and place this in a file:

specify { puts 1; expect(1).to eq(1) }
specify { puts 2; expect(1).to eq(1) }

Then 1 and 2 will be printed in a random order for each run of turbo_tests (without --seed given), yet no seed is shown in 2.2.4.

If turbo_tests doesn't display the random seed and requires that one be specified explicitly, then things are left in a surprising (to me) situation. With a test suite configured to use a random order:

• Running rspec requires no --seed value, and will print the random seed used for a test run.
• turbo_tests (2.2.4) will still run in a random order, but it won't show the seed used unless --seed is given.

That feels incorrect to me? I understand there are other bugs being addressed by the change in behavior here, but it effectively means that --seed must be given solely to make the seed value display: if --seed isn't given as a param, it is still random but the default behavior provides no automatic, direct way of reproducing an ordering that was used in the event of an order-dependent test failure.

As a user I think this makes turbo_tests harder to use as a drop-in for rspec: a dev cannot simply use turbo_tests over rspec, they must now ensure that --seed $RANDOM or an equivalent is specified in both automated tooling (e.g. CI) and in all their own command runs, ensure it's part of their aliases, etc., or do something custom to print the seed value at the start/end of the tests.

Could turbo_tests read the RSpec config, and use that to determine whether to act as if a --seed has been given?

[aprescott]

Could turbo_tests read the RSpec config, and use that to determine whether to act as if a --seed has been given?

To expand on that a bit: basically if config.order is :random then act as if turbo_tests was given --seed $RANDOM (or similar, I'm not sure what RSpec itself does in Ruby for example). That would allow turbo_tests to presumably fix the seed for each parallel run, printing the value before and after, preserving the expected behavior.

[ilyazub]

The actual issue/confusion with TurboTests is that it doesn't print the Randomized with seed 36736 line from output of the each of RSpec processes in TurboTests::Runner#handle_message. I'll try to fix it.

Then 1 and 2 will be printed in a random order for each run of turbo_tests (without --seed given), yet no seed is shown in 2.2.4.

Could turbo_tests read the RSpec config, and use that to determine whether to act as if a --seed has been given?

I see, thank you. --seed argument for turbo_tests is used for all rspec processes. Contrary, the config.order = :random is used to set the random seed per rspec process.

[aprescott]

The actual issue/confusion with TurboTests is that it doesn't print the Randomized with seed 36736 line from output of the each of RSpec processes in TurboTests::Runner#handle_message.

For what it's worth, this is a slightly different description of an issue/confusion than the one I have. I think there are at least 2 or 3 related things involved: config.order = :random being respected, that turbo_tests would pick a fixed seed used for each RSpec process, and that any seed information is printed.

If the seed is intended to be random per process by design, then there really needs to be a way to have access to that information. Without a printed seed value, then a randomized test run is much harder to reproduce, even when using the exact same subprocess command that turbo_tests ran.

Printing the seed value for each subprocess would certainly help with reproducing random-order test failures, even if there are different seeds for each subprocess. Since there are N separate seed values, I'm not too sure what that actually looks like in terms of command output.

--seed argument for turbo_tests is used for all rspec processes. Contrary, the config.order = :random is used to set the random seed per rspec process.

Indeed. I understand that difference, but what I was imagining is something like:

1. Read the RSpec configuration value of config.order.
2. If config.order is set to :random, then act as if --seed SOMERANDOMVALUE were given to turbo_tests. Use the seed value for each subprocess explicitly, so they all agree.
3. Since turbo_tests will act as if --seed were given, then it will print the seed in its output.

(It would also be great to ensure that --seed shows up in separate commands printed when using turbo_tests's --verbose output. I'm not able to check whether that currently happens right now, apologies if it does already!)

Checking config.order in this way would seem to avoid the need to manually give a random --seed argument to turbo_tests. It would make each subprocess use the same seed value of N, and the output formatter would print the resulting seed (the same way it would do if --seed were given).

[aprescott]

Could turbo_tests read the RSpec config, and use that to determine whether to act as if a --seed has been given?

I've opened a PR that implements this idea: https://github.com/serpapi/turbo_tests/pull/63.