test-utils Tests running in a GH action sharing a nuxt server a bit faster

Describe the feature

In my current Nuxt project I have 6 test files each testing about 8-10 endpoints, each file currently spins up its own nuxt instance making the entire action require about 10 minutes to run, and vitest itself taking 8m 46s

So I tweaked my GH action to instead run a nuxt server in an action in the background, and then sleep for 10 seconds to make sure it completes its run

  - name: Start Server
    run: pnpm run dev:test &
  - name: Sleep for 10 seconds
    run: sleep 10

Update: sleep no longer needed

Now instead of vitest taking 8 minutes 46 seconds, it finished in 18 seconds.

You can find the entire action file here

I found this pretty amazing so I thought I'd share starting with an issue with the questions:

Is there any way to improve how I'm doing this?
Where should we document this so the community can easily find it?

Very happy to provide a PR anywhere to help share this strategy

BTW shot out to @tobiasdiez for Making this possible

Additional information

[X] Would you be willing to help implement this feature?
[X] Could this feature be implemented as a module?

Final checks

[X] Read the contribution guide.
[X] Check existing discussions and issues.

Aug 24 '24 02:08 acidjazz

That's an interesting observation; thanks for sharing!

Unfortunately, I am not able to set up your project to run locally (Prisma related). It would be great if you could set up a minimal reproduction.

On the other hand, I suggest you could try the following two approaches:

Move the server start to a global setup file, so you don't need an extra step in your action: https://vitest.dev/config/#globalsetup
Try disabling isolation: https://vitest.dev/config/#pooloptions-threads-singlethread - so all your tests files shared a same instance (where you will then need to make sure your tests are side-effects free)
- If you do this without a problem, optionally you might opt-in concurrency for your tests to boost the speed: https://vitest.dev/api/#describe-concurrent

Aug 24 '24 14:08 antfu

Move the server start to a global setup file, so you don't need an extra step in your action: https://vitest.dev/config/#globalsetup

I wasnt able to get globalSetup working, here is what I tried:

vitest.config.ts

...
  test: {
    globalSetup: './test/globalSetup.ts',
...

in ./test/globalSetup.ts i tried

import { setup as nuxtSetup } from '@nuxt/test-utils'

export default async function setup() {
  await nuxtSetup()
}

(i tried both named and default)

and I get

Error: Vitest failed to access its internal state.

One of the following is possible:
- "vitest" is imported directly without running "vitest" command
- "vitest" is imported inside "globalSetup" (to fix this, use "setupFiles" instead, because "globalSetup" runs in a different context)
- Otherwise, it might be a Vitest bug. Please report it to https://github.com/vitest-dev/vitest/issues

 ❯ getWorkerState ../node_modules/.pnpm/[email protected]_@[email protected]_@[email protected][email protected][email protected]/node_modules/vitest/dist/chunks/utils.Ck2hJTRs.js:5:11
 ❯ getTestFile ../node_modules/.pnpm/[email protected]_@[email protected]_@[email protected][email protected][email protected]/node_modules/vitest/dist/chunks/vi.fiQ7lMRF.js:523:17
 ❯ createExpect ../node_modules/.pnpm/[email protected]_@[email protected]_@[email protected][email protected][email protected]/node_modules/vitest/dist/chunks/vi.fiQ7lMRF.js:466:20
 ❯ ../node_modules/.pnpm/[email protected]_@[email protected]_@[email protected][email protected][email protected]/node_modules/vitest/dist/chunks/vi.fiQ7lMRF.js:526:22
 ❯ ModuleJob.run ../node:internal/modules/esm/module_job:222:25
 ❯ ModuleLoader.import ../node:internal/modules/esm/loader:316:24
 ❯ ViteNodeRunner.interopedImport ../node_modules/.pnpm/[email protected]_@[email protected][email protected]/node_modules/vite-node/dist/client.mjs:421:28
 ❯ ViteNodeRunner.directRequest ../node_modules/.pnpm/[email protected]_@[email protected][email protected]/node_modules/vite-node/dist/client.mjs:280:24
 ❯ ViteNodeRunner.cachedRequest ../node_modules/.pnpm/[email protected]_@[email protected][email protected]/node_modules/vite-node/dist/client.mjs:206:14
 ❯ ViteNodeRunner.dependencyRequest ../node_modules/.pnpm/[email protected]_@[email protected][email protected]/node_modules/vite-node/dist/client.mjs:259:12

Aug 24 '24 23:08 acidjazz

Try disabling isolation: https://vitest.dev/config/#pooloptions-threads-singlethread - so all your tests files shared a same instance (where you will then need to make sure your tests are side-effects free)

When I do this I get prisma errors which I believe are when they are trying to create the same user at the exact same time:

FAIL  ../test/cartridge.test.ts > /api/cartridge > post /api/cartridge - create a cartridge
PrismaClientKnownRequestError:
Invalid `prisma.user.upsert()` invocation:


Unique constraint failed on the constraint: `users_email_key`
 ❯ _n.handleRequestError ../node_modules/.pnpm/@[email protected][email protected]/node_modules/@prisma/client/runtime/library.js:121:7749
 ❯ _n.handleAndLogRequestError ../node_modules/.pnpm/@[email protected][email protected]/node_modules/@prisma/client/runtime/library.js:121:7057
 ❯ _n.request ../node_modules/.pnpm/@[email protected][email protected]/node_modules/@prisma/client/runtime/library.js:121:6741
 ❯ l ../node_modules/.pnpm/@[email protected][email protected]/node_modules/@prisma/client/runtime/library.js:130:9355
 ❯ Module.createUser utils/user.ts:11:10
      9|   let user: User | null = null
     10|
     11|   user = await prisma.user.upsert({
       |          ^
     12|     where: { email: info.email },
     13|     create: {
 ❯ userFromEmail ../test/auth.ts:38:28
 ❯ Module.actingAs ../test/auth.ts:44:16
 ❯ ../test/cartridge.test.ts:12:28

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Serialized Error: { code: 'P2002', clientVersion: '5.18.0', meta: { modelName: 'User', target: 'users_email_key' }, batchRequestIdx: undefined }
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

to avoid this in the action or locally, i have to set this in vitest:

vitest.config.ts

...
  test: {
    poolOptions: {
      forks: {
        minForks: 1,
        maxForks: 1,
      },
    },
...

Created an issue in more detail here

Aug 24 '24 23:08 acidjazz

UPDATE I was able to swap out the nuxt dev & and sleep 10 actions with starting nuxt in the build:

The entire action now runs in 2m 11s

- name: Build and start server in the background
  run: pnpm run build:test:start:bg

here is build:test:start:bg in my package.json

"nuxt build --dotenv .env.test; node --env-file=.env.test .output/server/index.mjs &"

notice i had to use --env-file= which requires node v20+ so i also added actions/setup-node-v4

Updated action found here

Aug 25 '24 00:08 acidjazz

FWIW I run my e2e tests against a locally running wrangler environment. Playwright allows you to pass options to start up a webServer, but you could alternatively utilize a package like start-server-and-test to first spin up your test environment, and then run the tests once it's up.

Utilizing a package like this will also take care of shutting down the dev server when the tests complete or fail, etc.

Nov 29 '24 16:11 adamdehaven