oxc parser: Improve TS error story

Background

Boshen: I want to improve ts error story, but I have trouble differentiating ts syntax errors vs type errors in https://github.com/oxc-project/oxc/blob/main/tasks/coverage/snapshots/parser_typescript.snap

When it says:
Expect Syntax Error: tasks/coverage/typescript/tests/cases/compiler/ClassDeclaration24.ts
the error file tsc produces may contain type errors, which we should exclude.

I thought I could bisect https://github.com/microsoft/TypeScript/blob/main/src/compiler/diagnosticMessages.json and split the error codes, but the error codes seem pretty random.

Turning off noCheck doesn't really help, because it just turns off checker.ts, and checker.ts may produce syntax errors as well. e.g. class any {} (from ClassDeclaration24.ts) https://discord.com/channels/1079625926024900739/1080712024683708466/1380227952322674718

Progress

https://github.com/oxc-project/oxc/issues/11582#issuecomment-3008344585

The problem

In parser_typescript.snap, we sometimes get "Expect Syntax Error" lines, but it is not clear how to fix them.

How does this message appear:

We are using compiler and conformance fixtures from TypeScript tests inside tests/cases
Some test cases have a xxx.errors.txt file exists inside tests/baselines/reference
- This means this case is expected to fail
&& Our parser and semantic does NOT catch that error

So the message means: "TSC reported some errors, but OXC did not!"

However, xxx.errors.txt file can exist for several reasons:

The code is not valid JS or TS syntax
- e.g. if {, async class X {}, const a;, class any {}, class X { private private x = 1 }
The code is syntactically fine, but has only type errors
Invalid usage of CLI frags, compiler options, etc...

(IMPORTANT: this is just a snapshot of the results of TSC!)

Especially, case 2 is a problem for us. OXC does not support type inference, so the "Expect Syntax Error" message is confusing.

We would like to filter such cases out and improve coverage numbers.

Jun 10 '25 08:06 leaysgur

TSC have 2 main ways to get errors:

tsProgram.getSyntacticDiagnostics()
tsProgram.getSemanticDiagnostics()

The first case represents a critical syntax error, and OXC also wants to treat this as a parser error.

The second case is somewhat tricky, while TSC claims it's a semantic error, but we wants to treat many cases as parser errors instead. (e.g. class any {}, const a;)

Therefore, we need to look at all(!) errors reported by TSC and decide which ones should become parser errors in OXC.

https://github.com/microsoft/TypeScript/blob/main/src/compiler/diagnosticMessages.json

It cannot be easily categorized by just the code. We want some 2xxx series errors to be parser errors too, not just 1xxx series ones. And purely semantic errors are shown in other code ranges too...

Brief summary of what I have been looking into these days... (The full article (in Japanese) is available here)

There are 2,119 entries in diagnosticMessages.json for now
- However, about 130 of them seems unused
1,343/2,119 items are marked as an error category
- Items in the suggestion category—such as refactoring message in the editor—can be excluded
And also I realized it isn't necessary to inspect all 1,343 items
parser_typescript.snap targets the compiler and conformance tests only
Therefore, we only need to examine the error-category items listed in the .errors.txt files referenced by those tests
That reduces total amounts to 950 items, and their code distribution is as follows:

🍀 Found 700 error diagnostics are used for compiler tests.
- 1xxx: 214
- 2xxx: 331
- 4xxx: 10
- 5xxx: 26
- 6xxx: 22
- 7xxx: 34
- 8xxx: 18
- 9xxx: 19
- 17xxx: 12
- 18xxx: 14

🍀 Found 695 error diagnostics are used for conformance tests.
- 1xxx: 247
- 2xxx: 327
- 4xxx: 12
- 5xxx: 12
- 6xxx: 8
- 7xxx: 22
- 8xxx: 20
- 9xxx: 2
- 17xxx: 15
- 18xxx: 30

🍀 Found 950 error diagnostics are used for compiler|conformance tests.

Each TSC components(src/compiler/*.ts) reference which error codes gives the following:

🍀 Found 29 error diagnostics are used in "src/compiler/binder.ts".
- 1xxx: 18
- 2xxx: 7
- 5xxx: 1
- 7xxx: 2
- 18xxx: 1

🍀 Found 905 error diagnostics are used in "src/compiler/checker.ts".
- 1xxx: 258
- 2xxx: 502
- 4xxx: 20
- 5xxx: 7
- 6xxx: 15
- 7xxx: 39
- 8xxx: 13
- 17xxx: 12
- 18xxx: 39

🍀 Found 38 error diagnostics are used in "src/compiler/commandLineParser.ts".
- 1xxx: 3
- 5xxx: 16
- 6xxx: 12
- 8xxx: 1
- 17xxx: 2
- 18xxx: 4

🍀 Found 7 error diagnostics are used in "src/compiler/executeCommandLine.ts".
- 5xxx: 6
- 6xxx: 1

🍀 Found 3 error diagnostics are used in "src/compiler/moduleNameResolver.ts".
- 2xxx: 2
- 6xxx: 1

🍀 Found 78 error diagnostics are used in "src/compiler/parser.ts".
- 1xxx: 57
- 2xxx: 7
- 8xxx: 3
- 17xxx: 7
- 18xxx: 4

🍀 Found 172 error diagnostics are used in "src/compiler/program.ts".
- 1xxx: 77
- 2xxx: 15
- 5xxx: 35
- 6xxx: 16
- 7xxx: 2
- 8xxx: 15
- 17xxx: 3
- 18xxx: 9

🍀 Found 2 error diagnostics are used in "src/compiler/programDiagnostics.ts".
- 2xxx: 2

🍀 Found 67 error diagnostics are used in "src/compiler/scanner.ts".
- 1xxx: 64
- 6xxx: 2
- 18xxx: 1

🍀 Found 18 error diagnostics are used in "src/compiler/transformers/declarations.ts".
- 2xxx: 2
- 4xxx: 4
- 5xxx: 1
- 6xxx: 2
- 7xxx: 1
- 9xxx: 8

🍀 Found 115 error diagnostics are used in "src/compiler/transformers/declarations/diagnostics.ts".
- 4xxx: 87
- 9xxx: 28

🍀 Found 2 error diagnostics are used in "src/compiler/tsbuildPublic.ts".
- 6xxx: 2

🍀 Found 8 error diagnostics are used in "src/compiler/utilities.ts".
- 1xxx: 1
- 2xxx: 3
- 5xxx: 1
- 7xxx: 3

🍀 Found 3 error diagnostics are used in "src/compiler/utilitiesPublic.ts".
- 6xxx: 3

This confirms that we can't decide based solely on the error code

I'd like to keep exploring for a while whether there's a better approach than enumerating all 950 messages and adding them to an exclusion list one-by-one...

But it may possibly not be found.

https://github.com/microsoft/TypeScript/issues/52011

Jun 13 '25 04:06 leaysgur

We were looking for a declarative way to generate the list of error codes that our coverage tests should ignore.

I found that we cannot infer the real intention from the code value alone, so I tried working backward from where the codes are used.

But after going through TSC's Diagnostics and its baseline tests, I concluded that no declarative method exists...

The main problem is that checker.ts (and therefore getSemanticDiagnostics()) does not clearly separate the errors we care about from the ones we do not. (obviously 😅)

Searching for (parse|grammar)ErrorXxx() looked promising, but it would still miss many cases. For example, it will not catch class any {}, I gave up that idea.

Therefore, we will need to review about 950 error codes by hand and build an ignore list. Even with helper scripts, we would still need to check roughly 800 codes.

And updates may be rare, but keeping the list also be a maintenance burden?

Or, it may sound drastic, how about to stop using the TSC tests as fixtures for parser? These baseline tests are only snapshots for TSC's compiler output.

Most .errors.txt files contain several error codes, and many test cases include multiple files, which does not help.

Instead, we could use babel-parser/test/fixtures/typescript and align with babel-ts or @typescript-eslint/parser throw or NOT? (At the very least, Prettier would likely be happy?)

@Boshen What do you think? (Please correct me if this is that you already considered, or I'm overlooking something fundamental. 🙇🏻)

Jun 16 '25 09:06 leaysgur

Therefore, we will need to review about 950 error codes by hand and build an ignore list. Even with helper scripts, we would still need to check roughly 800 codes.

I assume it is way less if we keep all errors from parser.ts and binder.ts? The only error codes I need to manually check are from checker.ts

And updates may be rare, but keeping the list also be a maintenance burden?

You can write a script that greps parser.ts, binder.ts and checker.ts, get all the errors codes and save them into a file. We then keep a manually updated blacklist for type errors.

We run this script every time typescript is updated.

Jun 16 '25 11:06 Boshen

Thank you for your confirmation.

You can write a script that greps parser.ts, binder.ts and checker.ts

In the TSC codebase, each component's —such as parser.ts—all depend on a single "giant barrel export" that imports everything. This made me want to give up at first. 😓

But this morning, I came up with a different idea..., so I'll explore it further again.

Jun 17 '25 01:06 leaysgur

Thank you for your confirmation.

You can write a script that greps parser.ts, binder.ts and checker.ts

In the TSC codebase, each component's —such as parser.ts—all depend on a single "giant barrel export" that imports everything. This made me want to give up at first. 😓

But this morning, I came up with a different idea..., so I'll explore it further again.

https://github.com/ast-grep/ast-grep Diagnostics.xxx ? e.g. Diagnostics.Type_alias_name_cannot_be_0

Jun 17 '25 02:06 Boshen

ast-grep/ast-grep Diagnostics.xxx ? e.g. Diagnostics.Type_alias_name_cannot_be_0

This is exactly what I tried to produce this in my past comment.

🍀 Found 78 error diagnostics are used in "src/compiler/parser.ts".

But this can only find Diagnostics.xxx directly written in parser.ts.

Imported modules from _namespaces/ts.ts may also contain Diagnostics.xxx, and I'm trying to figure this out now.

Jun 17 '25 03:06 leaysgur

But this can only find Diagnostics.xxx directly written in parser.ts. Imported modules from _namespaces/ts.ts may also contain Diagnostics.xxx, and I'm trying to figure this out now.

To address this issue, I defined entry points for each component and bundled the code from there.

parser: createSourceFile()
binder: bindSourceFile()
checker: getTypeChecker()

The plan is to use rollup+esbuild with tree-shaking to more robustly retrieve only the Diagnostics actually used in the code. (I also tried rolldown, but it produced different results...🤔)

Code

import { rollup } from "rollup";
import virtual from "@rollup/plugin-virtual";
import esbuild from "rollup-plugin-esbuild";
import { parse, Lang } from "@ast-grep/napi";

const parserCode = await bundleWithTreeshake(`
import { createSourceFile } from "./src/compiler/parser.ts";
const ast = createSourceFile("dummy.ts", code, 99);
`);

// NOTE: Parser uses Scanner internally
const parserUsedDiagnostics = extractErrorDiagnostics(parserCode);
console.log(
  "🍀",
  parserUsedDiagnostics.size,
  "diagnostics used in the scanner+parser",
);

// ---

// Bundle the code:
// - to resolve `import` inside the each component
// - to drop unused code
async function bundleWithTreeshake(entry: string) {
  const bundle = await rollup({
    input: "entry",
    output: { format: "esm" },
    treeshake: {
      moduleSideEffects: false,
      preset: "smallest",
      propertyReadSideEffects: false,
    },
    plugins: [virtual({ entry }), esbuild({})],
    onLog: (level, log, handler) => {
      // This often happens with TSC...
      if (log.code !== "CIRCULAR_DEPENDENCY") handler(level, log);
    },
  });
  const {
    output: [{ code }],
  } = await bundle.generate({});
  bundle.close();

  return code;
}

function extractErrorDiagnostics(code: string) {
  const sgRoot = parse(Lang.TypeScript, code).root();
  const diagDecl = sgRoot.find("const Diagnostics = $$$")!;
  const diagCalls = diagDecl.findAll(
    "diag($CODE, DiagnosticCategory.Error, $_, $MESSAGE)",
  );
  const diagCallsWithOpts = diagDecl.findAll(
    "diag($CODE, DiagnosticCategory.Error, $_, $MESSAGE, $$$)",
  );

  const errorDiagnostics = new Map<number, string>();
  for (const diagCall of [...diagCalls, ...diagCallsWithOpts]) {
    const code = Number(diagCall.getMatch("CODE")!.text());
    const message = diagCall.getMatch("MESSAGE")!.text();
    errorDiagnostics.set(code, message);
  }

  return errorDiagnostics;
}

And the results:

🍀 151 diagnostics used in the scanner+parser
🍀 28 diagnostics used in the binder
🍀 904 diagnostics used in the checker
🍀 1083 diagnostics used in the scanner+parser, binder and checker

It appears that binder depends on parser, and checker also depends on both binder and parser. Therefore, These numbers exclude that dependencies.

What these numbers reveal:

The results remain largely unchanged when simply sg-ing files individually
However, the total count exceeds the 950 appearing in the compiler|conformance tests!

Since latter was unexpected, so I investigated: "How many of these error codes appearing in the code are actually used in tests?"

The results were:

- parser: 36/151
- binder: 2/28
- checker: 178/904

N/M = M error codes are found in the code, but N are not tested.

(It isn't clear whether this is simply because no test has been prepared, or if it is covered by other tests like transform, etc?)

I assume it is way less if we keep all errors from parser.ts and binder.ts?

From these results, as for parser and binder, only 141 error codes are likely used and also tested = actually in use.

Subtract this from the original 950, resulting in 950 - 141 = 809 kinds of codes to be reviewed as originating from checker.

(It didn't decrease much as expected!🥹)

Jun 18 '25 02:06 leaysgur

👍 Great work! Next step is to get these error codes into our conformance suite.

Jun 18 '25 03:06 Boshen

There's still the major task of creating a "list of codes to intentionally ignore due to suspected type errors" remaining. 😂

Unfortunately, while casually reviewing the 141 cases that I originally intended to exclude from checks, I came across several cases that clearly belong on the ignore list...(e.g. related to JSDoc)

Now that I can't simply trust everything, and considering that both 809 and 950 involve essentially the same level of effort, I'm leaning to review all 950 cases. 🤔

BTW, in the existing tests, we've been ignoring some "tests that specify an older version with @target:es5 and expect errors" but it appears these also requires updating.

I'm also looking into whether we can implement a systematic approach for this as well.

Jun 18 '25 08:06 leaysgur

After reconsideration, my revised plan is as follows:

Use ts-morph(or typescript) to parse all test cases from complier|comformance on the JS side
- Parse at the unit of @filename
- Specify OXC equivalent compilerOptions like target: esnext
- Ignore files such as .d.ts at this stage
- Also ignore cross-file analysis, options problem, module resolution, etc...
After parsing, collect errors using get(Syntactic|Semantic)Diagnostics()
- At this stage, discard any error codes that should be ignored
If no errors are present = These should be "Expect to parse" test
If errors remain = These should be "Expect Syntax Error" test
Use these two lists for coverage on the Rust side

I hope I haven't missed anything. 🙈

Jun 19 '25 07:06 leaysgur

👨🏻‍🍳 Now cooking... https://github.com/leaysgur/oxc-tsc-fixtures

UPDATE: Considering both the execution and maintenance costs of this script, as well as the cost of integrating numerous files into CI, we determined it wasn't the ideal approach, so abandoned it.

Jun 20 '25 14:06 leaysgur

My plans and progress:

[x] Check error codes in coverage tests
- Replace error_files with error_codes and check it in should_fail()
- #11858
[x] Fix to NOT apply this to semantic_typescript.snap
- #11902
[x] Create another handy viewer application...
- https://github.com/leaysgur/tsc-error_diagnostic_codes-viewer/
[x] Fix the current error_codes collection logic in typescript/meta.rs
- #11932
- #12159
[x] Review and fill more error codes
- [x] ⚠️ Last reviewed TS hash: 8ea03f88d039759018673f229addb87f579f326c(at Nov 10), may review diffs already completed range
- [x] 1xxx(357) #15493
- [x] 2xxx(426)
  - #12510
  - #12606
  - #12794
  - #12827
  - #15492
- [x] 4xxx(21) #12459
- [x] 5xxx(34) #12385
- [x] 6xxx(24) #12311
- [x] 7xxx(40) #12067
  - [x] Follow up: #12259
- [x] 8xxx(26) #12034
- [x] 9xxx(21) #11989
- [x] 17xxx(18) #11989
- [x] 18xxx(34) #11933
[ ] Migrate init.ts to oxc repo
- Like just update-trasnformer-fixtures
Extract actionable issues

Jun 26 '25 12:06 leaysgur