`sequence.shuffle: true` does not respect the cache, runs test suites in random order

[nstepien]

Describe the bug

I want tests to run in random order, and I also want slower test suites to run first and quicker test suites to run last, but it seems impossible without implementing a custom sequencer.

With test suites/files being isolated, I don't see the point of also running the test suites/files in random order.

Reproduction

https://stackblitz.com/edit/vitest-dev-vitest-c5wvvm?file=test%2Fsuite1.test.ts,test%2Fslow.test.ts,test%2Fsuite2.test.ts,vite.config.ts,test%2Fsuite0.test.ts&initialPath=vitest/

1. run npm run test:run to build the cache
2. try npm run test:run multiple times, you'll see the test suites run in random order
3. set shuffle to false, run npm run test:run, notice that test/slow.test.ts starts first

System Info

System:
    OS: Windows 10 10.0.22621
    CPU: (64) x64 AMD Ryzen Threadripper 3970X 32-Core Processor
    Memory: 17.54 GB / 31.86 GB
  Binaries:
    Node: 20.2.0 - C:\Program Files\nodejs\node.EXE
    npm: 9.6.7 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Edge: Spartan (44.22621.1778.0), Chromium (114.0.1823.43)
    Internet Explorer: 11.0.22621.1
  npmPackages:
    @vitejs/plugin-react: ^4.0.0 => 4.0.0
    @vitest/coverage-v8: ^0.32.0 => 0.32.0
    vite: ^4.3.5 => 4.3.9
    vitest: ^0.32.0 => 0.32.0

Used Package Manager

npm

Validations

• [X] Follow our Code of Conduct
• [X] Read the Contributing Guidelines.
• [X] Read the docs.
• [X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• [X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• [X] The provided reproduction is a minimal reproducible example of the bug.

[stackblitz[bot]]

Start a new pull request in StackBlitz Codeflow.

• vitest-dev-vitest-c5wvvm (Open reproduction)

[sheremet-va]

sequence.shuffle also runs test files in random order, this is by design. You should use a custom sequencer if you want to run only tests inside files at random - or use describe.shuffle syntax where you need it to be random.

[nstepien]

this is by design.

Why is it? Test files are isolated and shouldn't affect each others.

Implementing a custom sequencer for something that should IMO be the default behavior isn't convenient.

[sheremet-va]

this is by design.

Why is it? Test files are isolated and shouldn't affect each others.

You can disable isolation with --no-isolate flag.

[nstepien]

I never said I wanted to disable test isolation.

[sheremet-va]

I never said I wanted to disable test isolation.

And still, it is possible to disable isolation, so your argument about isolation is not entirely correct.

[nstepien]

Are you suggesting that because isolation can be disabled, that test suites should be run in random order? So wouldn't it make sense to run test suites in completion time order when isolation is enabled?

[sheremet-va]

This is documented behaviour: https://vitest.dev/config/#sequence-shuffle

Vitest usually uses cache to sort tests, so long running tests start earlier - this makes tests run faster. If your tests will run in random order you will lose this performance improvement, but it may be useful to track tests that accidentally depend on another run previously.

It is absolutely possible to affect other test files by using FS operations for example.

shuffle option randomizes everything. If you don't want to randomize test files, use the base sequencer:

import { BaseSequencer } from 'vitest/node'
export default defineConfig({
  test: {
    sequence: {
      sequencer: BaseSequencer,
      shuffle: true,
    }
  }
}) 

[sheremet-va]

On the other hand, it might be a good idea to provide an option to configure this:

export default defineConfig({
  test: {
    sequence: {
      shuffle: {
        files: false,
        tests: true
      },
    }
  }
})