Vitest runs tests 3x slower than Jest with threads: true (default) setting

[dmLinker]

Describe the bug

With the latest Vitest/Vite dependencies, Vitest is ~3 times slower than Jest in https://github.com/EvHaus/jest-vs-jasmine (more or less like a typical front-end repo) with the threads: true (default) setting.

Benchmark 2: yarn workspace jest test
  Time (mean ± σ):     34.704 s ±  2.548 s    [User: 91.246 s, System: 16.712 s]
  Range (min … max):   32.017 s … 41.447 s    10 runs
 
Benchmark 3: yarn workspace vitest test --threads=true
  Time (mean ± σ):     99.077 s ±  2.015 s    [User: 185.493 s, System: 40.303 s]
  Range (min … max):   97.352 s … 103.406 s    10 runs

Related comment https://github.com/vitest-dev/vitest/issues/229#issuecomment-1003235680. That issue was closed but threads: true is still 3-4 slower than Jest after the issue was closed and the repro dependencies updated).

The issue was raised in Discord chat as well.

With threads: false Vitest is ~2 times faster, but the cost of it - there's no real isolation. With threads: false it's unfortunately not even close what Jest is doing in terms or resetting state, so not even sure if makes sense to compare threads: false with Jest (maybe threads: false should be compared with uvu or other "run-once-and-forget" runners with no real isolation). In a variety of projects that we have worked on, even with pure stateless components, the proper isolation is a super important concern. Especially in watch mode where introducing some unwanted state/mocking or polluting global/process state is simply an expected thing to happen due to the nature of various incremental changes (that are not always good or final) that simply break not properly isolated watch mode.

Reproduction

git clone https://github.com/EvHaus/jest-vs-jasmine

yarn

(if required, install hyperfine)

hyperfine --warmup 1 'yarn workspace jest test' 'yarn workspace vitest test --threads=true'

System Info

System:
    OS: macOS 11.2.3
    CPU: (16) x64 Intel(R) Xeon(R) W-2140B CPU @ 3.20GHz
    Memory: 25.41 MB / 32.00 GB
    Shell: 3.2.57 - /bin/bash
  Binaries:
    Node: 16.13.1 - ~/.nvm/versions/node/v16.13.1/bin/node
    Yarn: 3.1.1 - ~/.yarn/bin/yarn
    npm: 8.1.2 - ~/.nvm/versions/node/v16.13.1/bin/npm
    Watchman: 4.9.0 - /usr/local/bin/watchman
  Browsers:
    Chrome: 97.0.4692.71
    Firefox Developer Edition: 61.0
    Safari: 14.0.3

Used Package Manager

npm

Validations

• [X] Follow our Code of Conduct
• [X] Read the Contributing Guidelines.
• [X] Read the docs.
• [X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• [X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• [X] The provided reproduction is a minimal reproducible example of the bug.

[cbxp]

Vitest is also about 2-3x slower than @web/test-runner on the same project.

But also, it reports the time incorrectly: when it says that tests ran for 7 seconds, the time utility reports it was actually 25s of user time.

[Demivan]

@cbxp User time will be higher because Vitest is multi-threaded. User time is the time processor spent running a process. If you have multiple cores, user time will be higher for multi-threaded apps. Because it aggregates all the work that all cores did.

[black7375]

One of the reasons why Jest is fast is cache.

I added yarn workspace jest test --no-cache to the example of jest-vs-jasmine. (Vitest's --threads=true is default.)

>> hyperfine --warmup 1 'yarn workspace jasmine test' 'yarn workspace jest test' 'yarn workspace jest test --no-cache' 'yarn workspace jest test --shard' 'yarn workspace vitest test' 'yarn workspace vitest test --isolate=false'
Benchmark 1: yarn workspace jasmine test
  Time (mean ± σ):     10.317 s ±  1.826 s    [User: 12.119 s, System: 0.563 s]
  Range (min … max):    8.012 s … 13.697 s    10 runs
 
Benchmark 2: yarn workspace jest test
  Time (mean ± σ):     30.254 s ±  1.570 s    [User: 185.648 s, System: 9.170 s]
  Range (min … max):   28.122 s … 32.521 s    10 runs
 
Benchmark 3: yarn workspace jest test --no-cache
  Time (mean ± σ):     44.370 s ±  1.824 s    [User: 296.171 s, System: 10.601 s]
  Range (min … max):   42.537 s … 48.359 s    10 runs
 
Benchmark 4: yarn workspace jest test --shard
  Time (mean ± σ):     30.358 s ±  1.592 s    [User: 186.843 s, System: 9.127 s]
  Range (min … max):   28.141 s … 32.958 s    10 runs
 
Benchmark 5: yarn workspace vitest test
  Time (mean ± σ):     90.731 s ±  2.527 s    [User: 529.527 s, System: 36.283 s]
  Range (min … max):   87.438 s … 94.203 s    10 runs
 
Benchmark 6: yarn workspace vitest test --isolate=false
  Time (mean ± σ):     17.051 s ±  0.867 s    [User: 49.972 s, System: 2.401 s]
  Range (min … max):   15.555 s … 18.515 s    10 runs
 
Summary
  'yarn workspace jasmine test' ran
    1.65 ± 0.30 times faster than 'yarn workspace vitest test --isolate=false'
    2.93 ± 0.54 times faster than 'yarn workspace jest test'
    2.94 ± 0.54 times faster than 'yarn workspace jest test --shard'
    4.30 ± 0.78 times faster than 'yarn workspace jest test --no-cache'
    8.79 ± 1.58 times faster than 'yarn workspace vitest test'

---

Another benchmark. mocha's parallel looks pretty fast, too.

https://medium.com/dailyjs/javascript-test-runners-benchmark-3a78d4117b4

[pmdarrow]

My team in the process of migrating a large test suite (6073 tests) from jest to vitest. With jest, the full suite runs in 3m47s. With the latest vitest (0.18.0) it runs in 10m7s. We really want to continue with vitest but this kind of a slow down is making it challenging. We don't want to disable isolation, as it's crucial for reliability in a test suite this large. Is there any plan to improve the performance vitest, or is jest doing something fundamentally different that vitest won't be able to replicate? @antfu @sheremet-va @Demivan

[Demivan]

@pmdarrow I'm investigating switching to vm.runInNewContext from vm.runInThisContext. This, hopefully, will allow us to drop costly isolation using Workers. We will essentially have the same isolation as Jest.

But I did not have time to work on this recently (there is the war going on)

[sheremet-va]

@pmdarrow I'm investigating switching to vm.runInNewContext from vm.runInThisContext. This, hopefully, will allow us to drop costly isolation using Workers. We will essentially have the same isolation as Jest.

I don't think we can do it without getting the same problems Jest has, because VM has terrible support for ESM. I already tried it.

[sheremet-va]

My team in the process of migrating a large test suite (6073 tests) from jest to vitest. With jest, the full suite runs in 3m47s. With the latest vitest (0.18.0) it runs in 10m7s. We really want to continue with vitest but this kind of a slow down is making it challenging. We don't want to disable isolation, as it's crucial for reliability in a test suite this large. Is there any plan to improve the performance vitest, or is jest doing something fundamentally different that vitest won't be able to replicate? @antfu @sheremet-va @Demivan

You can get speed boost by disabling isolation. Bear in mind tho, that it will make globalThis object sharable between tests.

[Demivan]

After investigating it more and checking with @sheremet-va i don't think it can be implemented with current state of vm module. Fixing all the issues will mean getting all Jest bugs too.

Lets hope ShadowRealm implementation will be better then vm module.

[pmdarrow]

So is the summary that we're stuck with the performance where it's at right now, unless we want to sacrifice isolation? If so, I think this should be published in the docs somewhere. I know the docs say disabling isolation can improve performance, but it doesn't communicate that comparable tools (i.e. Jest) are 3x faster even with isolation. It would help to know up front why vite is slower by default and what is blocking it from becoming faster (vm module bugs?)

[antfu]

Vitest runs tests 3x slower than Jest

That is an incorrect statement. It really deps on what and how your tests are written. You can't say a man walks faster than cars without the context (in a traffic jam or something it might be true). The reason why this is happening commonly due to you are using some giant dependencies, where in isolation, they will be executed multiple times for each tests. Vitest is on-demand like Vite, while Jest will bundle the code, so in some scenario it would be more efficient than Vitest.

While indeed there is some space for optimization like pre-bundle the deps, or waiting for vm to support ESM. But in general, it's still trade-offs between speed and correctness. For performance and also for good practice, I would suggest to make the tests as "unit" as possible, or mocking dependencies whenever you can.

[dmLinker]

while Jest will bundle the code, so in some scenario it would be more efficient than Vitest.

Can you please elaborate a bit on how/where exactly Jest bundles anything? AFAIK Jest transformers are invoked lazily whenever a file is being loaded by other files during a test file execution.

it's still trade-offs between speed and correctness. For performance and also for good practice, I would suggest to make the tests as "unit" as possible, or mocking dependencies whenever you can.

The thing is that such "unit" tests (deeply mocked) tests are fast in both Jest and Vitest (the difference is really minor, and not always in favour of the same framework). However, in addition, Jest is isolating much better, so the trade-offs between speed and correctness case are relevant for Vitest only, while with Jest one can get both in this case.

Unfortunately the isolation with the use vm in Jest comes with the current ESM support limitations. Hopefully those are resolved in future so potentially both frameworks can benefit from it.

[thebuilder]

Is there anyway of debugging vitest performance? Maybe list what takes time on each run, or which dependencies are loaded? After switching a React Testing Library application (built with Vite) from Jest to Vitest, the test time CI (Azure DevOps) went from around 50s with code coverage using Jest, to 2 minutes without code coverage in Vitest. It could very well be related to dependencies we are using (like Material UI), but it's really hard to debug.

[Weetbix]

Like @thebuilder and @pmdarrow we have also experienced a 3x slowdown of tests after migrating to vitest from jest, our large CI unit test suite now takes 15 mins instead of 5 when running on GHA.

[antfu]

Ok, I can have a look at this. Can anyone set up a minimal repo (2~3 test files, or the less possible) of how the slowdown is being observed (as I personally do not face such a slowdown when doing unit testing or testing Vue)? Thanks

[EvHaus]

@antfu Maybe https://github.com/EvHaus/jest-vs-jasmine can help. It's more than 2~3 test files, but I think you can easily just strip out the test suites and reduce them. The latest benchmark shows:

   73.626s ± 0.308s for 'yarn workspace jest test'
   232.393s ± 0.457s for 'yarn workspace vitest test'

[sheremet-va]

Hm, looking at the repo, I wonder if the problem is in .module.less files 🤔 Currently Vitest doesn't process css files, but processes *.module.* by default (we have a PR to auto mock it in https://github.com/vitest-dev/vitest/issues/1512). We can start from that 👀

Disabling css with css: false and returning proxy for .module.less:

// plugin.js
{
      name: "remove-css",
      enforce: "post",
      transform(code, id) {
        if (id.endsWith(".module.less")) {
          const code = `
          export default new Proxy(
            {},
            {
              get(_, style) {
                return style;
              },
            }
          );`;
          return {
            code: code,
          };
        }
      },
    },

I have this result on M1 Air 8 GB:

Benchmark 1: yarn workspace jasmine test
  Time (mean ± σ):      3.719 s ±  0.052 s    [User: 3.476 s, System: 0.238 s]
  Range (min … max):    3.651 s …  3.838 s    10 runs
 
Benchmark 2: yarn workspace jest test
  Time (mean ± σ):     10.829 s ±  0.395 s    [User: 56.573 s, System: 6.929 s]
  Range (min … max):   10.561 s … 11.898 s    10 runs
 
Benchmark 3: yarn workspace jest test --shard
  Time (mean ± σ):     11.316 s ±  0.445 s    [User: 59.148 s, System: 7.274 s]
  Range (min … max):   10.771 s … 12.197 s    10 runs
 
Benchmark 4: yarn workspace vitest test
  Time (mean ± σ):     34.201 s ±  0.425 s    [User: 176.166 s, System: 32.009 s]
  Range (min … max):   33.561 s … 34.940 s    10 runs
 
Benchmark 5: yarn workspace vitest test --isolate=false
  Time (mean ± σ):      6.413 s ±  0.885 s    [User: 16.657 s, System: 1.793 s]
  Range (min … max):    6.076 s …  8.927 s    10 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
 
Summary
  'yarn workspace jasmine test' ran
    1.72 ± 0.24 times faster than 'yarn workspace vitest test --isolate=false'
    2.91 ± 0.11 times faster than 'yarn workspace jest test'
    3.04 ± 0.13 times faster than 'yarn workspace jest test --shard'
    9.20 ± 0.17 times faster than 'yarn workspace vitest test'

Maybe you should also benchmark --no-threads, btw?

Considering CSS, I don't see a lot of changes unfortunately. So It seems the problem lies with how tinypool fires up isolated workers.

[pmdarrow]

@antfu is https://github.com/EvHaus/jest-vs-jasmine sufficient for a reproduction or would you like it to be trimmed down further?

[antfu]

I trimmed a bit locally. Flow + Less is a bit less common tbh, but yes with your tests I can confirm it surfaced some bottleneck of Vitest, where I am still investigating.

[pmdarrow]

@antfu is there anything that we could do to help move this forward?

[phifa]

I am not sure my issue is related, but from time to time this happens, my fans are going off and cpu usage rises to 700%... Something going on with vitest... If this is related, how can I debug, mitigate this?

[pmdarrow]

@phifa I don't think that's related to this discussion - that's just vitest using as many cores as possible. If you don't want this behaviour, you can turn it off: https://vitest.dev/guide/features.html#threads.

[MarkLyck]

Same experience here. Jest ran my test-suite in about 3-5 minutes, after migrating to vitest it now takes 10-17 minutes 😞

I'm sure it would be a lot faster if I disabled threads, but it's not worth dealing with all the potential issues that comes with that.

I do have .less files in my project as well, but they are not called .module.less so I'm not sure if that would affect anything?

[sheremet-va]

Same experience here

I wonder, if enabling deps.fallbackCJS helps?

[ild0tt0re]

@dmLinker @pmdarrow We also want to use vitest as our modern test tool but for now we are using @swc/jest with good performance gain (3x)

[EvHaus]

FYI -- I updated my benchmarks to get rid of Flow, LESS and switched to more modern ways of doing things: TypeScript, Testing-Library, etc... and that had a drastic impact on the results.

Vitest is now consistently the fastest, with Jasmine just behind and Jest still consistently 4x slower than both (details).

NOTE: I'm using isolate: false in Vitest to ensure it performs at its best.

Summary
  'yarn workspace vitest test' ran
    1.17 ± 0.01 times faster than 'yarn workspace jasmine test'
    4.64 ± 0.03 times faster than 'yarn workspace jest test'
    4.64 ± 0.06 times faster than 'yarn workspace jest-swc test'
   10.11 ± 0.06 times faster than 'yarn workspace fastest-jest-runner test'

What this tells me is that there's some other external factor at play here. Perhaps a third-party library, or an extra preprocessor, or something.

Also interestingly, swc had 0 effect on the performance of jest in my case.

[antfu]

Glad to hear that, thanks for the update @EvHaus!

I think there is indeed some more space for us to optimize further.

[pmdarrow]

@EvHaus @antfu that's not an apples-to-apples comparison. Jest has fully isolated tests, so setting isolate: false is an unfair comparison. And it's clear most devs want their tests isolated.

[black7375]

@EvHaus As you can see from the benchmark above, there was a performance problem when isolate was true.

• https://github.com/vitest-dev/vitest/issues/579#issuecomment-1154571206
• https://github.com/vitest-dev/vitest/issues/579#issuecomment-1197227773

[thebuilder]

Is it even possible to use testing-library and JSDom without isolate: true? You can't use screen to getBy* from the DOM since it will most likely contain multiple render outputs at the same time.

[pmdarrow]

Speaking from experience, enabling isolate: false in a large test suite will cause all sorts of unexpected behaviour and flaky tests. In theory, it's possible, but if you add more than a couple developers on a project, it will get out of hand pretty quickly.