[Idea] Test Level (Error, Warn, Info)

[arpowers]

Clear and concise description of the problem

Hey there,

We are working on CI for a fully featured analytics app and are using Vitest extensively (very happy)..

However, some of the tests we are running are critical path, while others aren't. For example, we are running HTTP checks on services to check health, however, one service is always failing and restarting (due to a Playwright bug)... we'd like to issue a warning here rather than fail the suite.

I suggest a simple option for tests/suites to set a level similar to that in logging (error, warn, info, debug)

Suggested solution

Adding an option for test "level" to describe or test/it..

it(`tests something`, () => {
// test 
}, {
  timeout: 5000, level: 'warn'
})

Alternative

I realize we can name tests to allow them to be run selectively but that couples filename to run context.

Additional context

No response

Validations

• [X] Follow our Code of Conduct
• [X] Read the Contributing Guidelines.
• [X] Read the docs.
• [X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

[arpowers]

@sheremet-va @antfu this would also solve a separate issue with inline snapshots.

Often we want to use snapshots as a guide for more specific assertions, however, as "assertions" they always fail with timestamps (always different), generated IDs, etc...

Would be nice to print a warning only for failing snapshots instead of hard fail the suite.

[esquevin]

You should probably mock date / id generation to control their outputs rather than "accept" that your test is failing.

https://vitest.dev/guide/mocking.html

If we accept that a test "can" fail, we won't see when we broke the app. If your test is unreliable, either fix it or skip it.

[arpowers]

@esquevin thanks for the response, we mock a lot of stuff actually... the above idea would just help with speed+intent in different testing environments (e.g. CI, dev, vs prod testing)

[sheremet-va]

I think the idea is interesting. PR welcome!