Degraded Test Execution Time

[mikearnaldi]

Describe the bug

Reopening: https://github.com/vitest-dev/vitest/issues/1844

The proposed solution, to set maxConcurrency made the test run even slower.

See: https://github.com/Effect-TS/core/runs/7854754823?check_suite_focus=true

Reproduction

Same as before.

System Info

Same as before.

Used Package Manager

yarn

Validations

• [X] Follow our Code of Conduct
• [X] Read the Contributing Guidelines.
• [X] Read the docs.
• [X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• [X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• [X] The provided reproduction is a minimal reproducible example of the bug.

[sheremet-va]

The proposed solution, to set maxConcurrency made the test run even slower.

Didn't duration improved from 658 to 468? (Not 322, ofc, but not "slower" either 🤔 )

Setup/collect/test is accumulation of all test runs, so it might be big.

[mikearnaldi]

I was actually looking at the time spent in collections which seemed higher, yes overall it's faster than before but far away from what it was in 0.14.2

[Xa3lis]

I was actually looking at the time spent in collections which seemed higher, yes overall it's faster than before but far away from what it was in 0.14.2

Got the same problem, it's easily 10-15x slower than before (The setup phase was 200ms it's now 160s, 30-35s by thread)

[sheremet-va]

@mikearnaldi try enabling deps.fallbackCJS, I see some improvements in time on my machine.

[mikearnaldi]

@sheremet-va while it may be useful we will be migrating to esm only shortly

[sheremet-va]

@sheremet-va while it may be useful we will be migrating to esm only shortly

It doesn't affect your migration. A lot of packages have incorrect entries for ESM, so Vitest has to process it before running (more about it here). This takes a lot of time sometimes. fallbackCJS tries to find CJS entry instead to not fix developers mistakes. It flag was disabled by default some time ago.

If this is the case, Vitest can't help you with performance issues, until package authors fix their ESM entries.

[mikearnaldi]

Tested, doesn't seem to make a big difference. Also Effect is zero dependency and with perfectly working ESM modules so not sure that flag even counts in my case

[Xa3lis]

I managed to have decent performance by upgrading MSW and vite to their latest version, setup times are back to normal

[hmajid2301]

I managed to have decent performance by upgrading MSW and vite to their latest version, setup times are back to normal

This seemed to help on Gitlab CI seemed to speed up my test by about ~2mins (7 min -> 5 min).

Commit: https://gitlab.com/bookmarkey/gui/-/merge_requests/53/diffs?commit_id=3fce01c0f71dbdcb6c8d32012be815cb318fbfca

Old Pipeline: https://gitlab.com/bookmarkey/gui/-/jobs/3553652699 New Pipeline: https://gitlab.com/bookmarkey/gui/-/jobs/3553775928

[sheremet-va]

@mikearnaldi can you check the latest version? We had some performance releases.

[hmajid2301]

In my case this has not much of a difference its still taking around 14 minutes to run ~56 tests; https://gitlab.com/bookmarkey/gui/-/jobs/3700219999 Which is about the same it was taking before.

Edit: This was caused by something else

[AriPerkkio]

@hmajid2301 in your case one of the Svelte file imports is unable to tree shake imports. This seems to take most of the time.

import { FolderSolid } from "svelte-awesome-icons";

This ends ups importing all +2000 icons.

const __vite_ssr_import_0__ = await __vite_ssr_import__("/node_modules/.pnpm/[email protected]/node_modules/svelte-awesome-icons/PasteSolid.svelte");
const __vite_ssr_import_1__ = await __vite_ssr_import__("/node_modules/.pnpm/[email protected]/node_modules/svelte-awesome-icons/ScrewdriverSolid.svelte");
const __vite_ssr_import_2__ = await __vite_ssr_import__("/node_modules/.pnpm/[email protected]/node_modules/svelte-awesome-icons/VimeoVBrand.svelte");
...

I can't spot any incorrect configurations from the published package. https://www.npmjs.com/package/svelte-awesome-icons?activeTab=explore ~~Maybe some plugin set by svelte-kit is doing that - transforming it early to CJS for SSR?~~

[hmajid2301]

Ahh thank you raising, I'm not super familiar with frontend development!

[hmajid2301]

@AriPerkkio Where did you find that file that shows all the imports ?

[AriPerkkio]

It's the gui/node_modules/svelte-awesome-icons/index.js. But as said, the package itself seems fine. It's just Vitest which ends up processing all the files imported in that entrypoint.

In your case it's possible to reduce the test time by changing the import statements a bit. For example with the FolderSolid:

-import { FolderSolid } from "svelte-awesome-icons";
+import FolderSolid from "svelte-awesome-icons/FolderSolid.svelte";

Maybe some custom resolver in test.alias could do this automatically.

[hmajid2301]

Yeh that seems to have done down from 14mins -> 4mins after this commit: https://gitlab.com/bookmarkey/gui/-/merge_requests/75/diffs?commit_id=0eb1a6917cdd9b226937ecdda1571c29f3c4e3a7

Thanks @AriPerkkio

[sheremet-va]

@mikearnaldi is it still an issue? I see that you halved the amount of tests that you had.

Side note: I was able to make your tests x2 times faster by disabling isolation in effect repo (since you don't rely on any global state):

export default defineConfig({
  test: {
    isolate: false,
  },
})

[tlebel]

in your case one of the Svelte file imports is unable to tree shake imports. This seems to take most of the time.

import { FolderSolid } from "svelte-awesome-icons";

This ends ups importing all +2000 icons.

const __vite_ssr_import_0__ = await __vite_ssr_import__("/node_modules/.pnpm/[email protected]/node_modules/svelte-awesome-icons/PasteSolid.svelte");
const __vite_ssr_import_1__ = await __vite_ssr_import__("/node_modules/.pnpm/[email protected]/node_modules/svelte-awesome-icons/ScrewdriverSolid.svelte");
const __vite_ssr_import_2__ = await __vite_ssr_import__("/node_modules/.pnpm/[email protected]/node_modules/svelte-awesome-icons/VimeoVBrand.svelte");
...

@AriPerkkio How did you spot this? Trying to debug similar issue where 1 particular test has nothing special, yet, it taking over 15 sec to run. I see the transform/collect is really the bulk of the work:

Duration 16.55s (transform 13.38s, setup 925ms, collect 13.19s, tests 1.85s, environment 294ms, prepare 81ms)

Wondering how I can inspect what is going on in vitest? Vite has inspect that seems could help here but I don't see how I can use it with vitest

Thanks!

[AriPerkkio]

I think it showed up when using DEBUG=vite-node:* envrionment variable.

Vite has inspect that seems could help here but I don't see how I can use it with vitest

vite-plugin-inspect works with Vitest as well. I've been using it with the build mode: https://github.com/antfu/vite-plugin-inspect?tab=readme-ov-file#build-mode.

[sheremet-va]

I am closing this issue because there was no response from the author, and the reproduction is no longer available. We also released a lot of performance improvements since the issue was created, so I hope it was enough.