feat: attest integration

[hi-ogawa]

Description

• closes https://github.com/vitest-dev/vitest/issues/5857

Based on https://github.com/vitest-dev/vitest/pull/6852, now I moved all code to core.

summary

Add new assertions:

• toMatchTypeSnapshot/toMatchTypeInlineSnapshot
• toMatchTypeErrorSnapshot/toMatchTypeErrorInlineSnapshot
• toMatchTypeCompletionSnapshot/toMatchTypeCompletionInlineSnapshot

and a new option to "enable" them (without the option, assertions become no-op and no-error):

• vitest --attest
• or defineConfig({ test: { attest: true } })

Example on stackblitz https://stackblitz.com/edit/vitest-dev-vitest-ywts26?file=test%2Fbasic.test.ts

example code

test('inline', () => {
  expect(squared).toMatchTypeInlineSnapshot(`(n: number) => number`);

  expect(() =>
    // @ts-expect-error
    squared('foo')
  ).toMatchTypeErrorInlineSnapshot(
    `Argument of type 'string' is not assignable to parameter of type 'number'.`
  );

  expect(
    () =>
      // @ts-expect-error
      squared['a']
  ).toMatchTypeCompletionInlineSnapshot(`
    {
      "a": [
        "apply",
        "arguments",
      ],
    }
  `);
});

todo

• [x] implement assertions
  • [x] type
  • [x] errors
  • [x] completions
• [x] config to enable attest
  • [ ] support custom ATTEST_tsconfig
• [ ] doc
• [ ] polish
  • vitest
    • [ ] avoid new Error().stack for for non type assertion expect(xxx) (how?)
    • [ ] error handling for mis-usage, default behavior on CI etc...
    • [x] prompt install
    • [x] remove PrettyFormatSkipSnapshotError hack
  • attest
    • [x] programmatic API for precache
    • [ ] avoid cwd reliance on runtime
      • allow ATTEST_assertionCacheDir option https://github.com/arktypeio/arktype/blob/a5f7e4b2b857f2275e8ea03d9895d12db3076b42/ark/attest/config.ts#L149-L151
      • what to do with getFileKey? https://github.com/arktypeio/arktype/blob/776ce7e85becf7614b4ad4def65d0d90df9c908a/ark/attest/utils.ts#L5
    • [ ] avoid analyzing non type assertion expect(xxx) (can be later)
    • [ ] avoid fs and process on runtime (later for browser mode)
      • https://github.com/arktypeio/arktype/blob/a5f7e4b2b857f2275e8ea03d9895d12db3076b42/ark/attest/cache/getCachedAssertions.ts#L26
  • [ ] make it work on stackblitz https://stackblitz.com/edit/vitest-dev-vitest-ywts26?file=test%2Fbasic.test.ts
    • rerun broken? (their file system issue?)

tbd

• [ ] config , default behavior, etc...
• [ ] workspace support? (currently attest runs independently for each project with { attest: true })

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

• [x] It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
• [x] Ideally, include a test that fails without this PR but passes with it.
• [ ] Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.

Tests

• [ ] Run the tests with pnpm test:ci.

Documentation

• [ ] If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

• [x] Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

[netlify[bot]]

✅ Deploy Preview for vitest-dev ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	2ee32da6f179d44ec6dd1be242523c0e25071e59
🔍 Latest deploy log	https://app.netlify.com/sites/vitest-dev/deploys/67459f0632e1c40008947a19
😎 Deploy Preview	https://deploy-preview-6963--vitest-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[pkg-pr-new[bot]]

@vitest/coverage-istanbul

npm i https://pkg.pr.new/@vitest/coverage-istanbul@6963

@vitest/expect

npm i https://pkg.pr.new/@vitest/expect@6963

@vitest/mocker

npm i https://pkg.pr.new/@vitest/mocker@6963

@vitest/browser

npm i https://pkg.pr.new/@vitest/browser@6963

@vitest/coverage-v8

npm i https://pkg.pr.new/@vitest/coverage-v8@6963

@vitest/pretty-format

npm i https://pkg.pr.new/@vitest/pretty-format@6963

@vitest/runner

npm i https://pkg.pr.new/@vitest/runner@6963

@vitest/snapshot

npm i https://pkg.pr.new/@vitest/snapshot@6963

@vitest/spy

npm i https://pkg.pr.new/@vitest/spy@6963

@vitest/ui

npm i https://pkg.pr.new/@vitest/ui@6963

@vitest/utils

npm i https://pkg.pr.new/@vitest/utils@6963

vite-node

npm i https://pkg.pr.new/vite-node@6963

vitest

npm i https://pkg.pr.new/vitest@6963

@vitest/web-worker

npm i https://pkg.pr.new/@vitest/web-worker@6963

@vitest/ws-client

npm i https://pkg.pr.new/@vitest/ws-client@6963

commit: 7d0bfd5

[mrazauskas]

Perhaps it would be useful to add more complex examples? I tried to play with a recursive type:

import { expect, test } from 'vitest';

type JsonValue = string | number | boolean | JsonObject | Array<JsonValue>;

interface JsonObject {
  [key: string]: JsonValue;
}

function stringify(input: JsonValue): string {
  return JSON.stringify(input);
};

test('stringify', () => {
  expect(stringify).toMatchTypeInlineSnapshot(`(input: JsonValue) => string`);
});

Do you think this snapshot is good?

I mean, it will fail if name of the argument gets changed, although that’s not really important. But it will pass if type is redefined into type JsonValue = Record<string, unknown>, although this is exactly the change one would like to catch. Did I miss something?

JSX components were mention in #5857, so here is an attempt:

interface ButtonProps {
  text: string;
  type?: "reset" | "submit";
}
 
function Button({ text, type }: ButtonProps) {
  return <button type={type}>{text}</button>;
}

test('Button', () => {
  expect(Button).toMatchTypeInlineSnapshot(`({ text, type }: ButtonProps) => Element`);
});

Good to see names of props, but if type would be marked required, that will not get caught. Also note that JSX.Element got stringified as Element. Namespace is lost. I have noticed this previously in tests of DefinetelyTyped libraries. For instance, Immutable.Map and the built-in Map both get stringified as Map by dtslint.

[hi-ogawa]

@mrazauskas Hi, first of all, thanks for testing out!

About what's printed in the snapshot, we've talked with arktype/attest author @ssalbdivad. If I remember correctly, that's what we can get from ts server api. Normally it should match with what you see when hovering on IDE, but vscode can somehow show namespace without getting stripped.

We're not familiar with this area, so mostly we'll need to delegate to what attest supports for now even if we integrate some features as Vitest builtin. I guess this feature will start as experimental and see if the representation makes sense to users in general.

[mrazauskas]

We're not familiar with this area

Fair enough. I mean.. Are you about to ship this feature knowing that you are not familiar with it?

[hi-ogawa]

In terms of API, I'm mostly only using writeAssertionData from @ark/attest, which analyzes xxx inside expect(xxx) and then output .attest/assertions/typescript.json file.

This in turn uses ts.typeChecker.typeToString API from typescript. https://github.com/arktypeio/arktype/blob/649f70eb2ec48b5203099469c6b59d0b5fe29260/ark/attest/cache/ts.ts#L208-L230 If the limitation is typescript API level, then Vitest cannot change anything for the snapshot though :thinking: