oxc-project
`oxc-parser` NPM package return AST as JS object, not JSON
At present, NPM package oxc-parser's parse* methods return an object including AST as JSON string.
I propose:
- Alter
parseSyncandparseAsyncto callJSON.parseon the JSON by default, so they return AST as a JS object. - Add an option
json: trueto return JSON string instead (as at present).
This is a breaking change, and will require bumping the package minor version (i.e. 0.21.0).
This is motivated by oxc-project/backlog#47.
@Boshen and I discussed on video call just now. He's given his 👍 to this.
@overlookmotel Supporting TypeScript interfaces for program also would be helpful or using as TypeScript object as const which returns RAW data as type
Hi @dalisoft. Sorry but I don't quite understand what you mean. I'm not much of a TypeScript user, so perhaps that's the reason. Could you possibly elaborate?
@overlookmotel
Input
const funcString =
/**
*
* @returns {{status: 'success' | 'error'}}
*/
(async (req, res) => {
const var1 = req.headers['X-Telemetry-ID'];
const var2 = req.cookies['X-User-ID'];
useEffect(() => {
let unload;
const initTelemetry = async () => {
await xUserConnect(req.params['x-user-id']);
({ unload } = await loadTelemetryApp(var1));
};
initTelemetry();
return unload;
}, [var1, var2]);
const result = await someExpensiveTask();
const confirm = await waitToConfirm(req, result);
const { cookies } = req;
const { send } = res;
const queries = req.queries;
const { compress } = req.queries;
const { check_zstd } = queries;
const values = [req.body];
const foo = {
bar: req.body.bar
};
const template = `${req.params.mailId}`;
await testMe(queries.elemId);
shouldSync(req.params);
await checkAuth(req.headers, req.cookies);
await itWorks(result, confirm);
itWorksSync(confirm);
await (async () => {
await someSQLTask();
})();
return { status: 'success' };
}).toString();
const oxcParserResult = await oxcParser.parseAsync(funcString, {
sourceType: 'unambiguous',
sourceFilename: `route-${i}.ts`,
preserveParens: true
});
const { errors, comments, program } = oxcParserResult;
// program is `string` at here, but as this issue propose is using `JS` instead of `JSON` string, i would like to expect `TypeScript types` too
Propose 1 (TypeScript types)
// This is for example, this interface/type could be improved
// with type-based properties like `body` or `properties`
interface ASTNode {
type:
| 'Program'
| 'ExpressionStatement'
| 'ArrowFunctionExpression'
| 'CallExpression' /* so on other types */;
start: number;
end: number;
body?: ASTNode[];
}
const { errors, comments, program } = oxcParserResult;
// program is `ASTNode` at here for better completion
Propose 2 (TypeScript as const)
ast-tree.ts
const result = {
type: 'Program',
start: 0,
end: 892,
sourceType: {
language: 'typescript',
moduleKind: 'module',
variant: 'standard',
alwaysStrict: false
},
hashbang: null,
directives: [],
body: [
{
type: 'ExpressionStatement',
start: 0,
end: 892,
expression: {
type: 'ArrowFunctionExpression',
start: 0,
end: 892,
expression: false,
async: true,
typeParameters: null,
params: {
type: 'FormalParameters',
start: 6,
end: 16,
kind: 'ArrowFormalParameters',
items: [
{
type: 'FormalParameter',
start: 7,
end: 10,
decorators: [],
pattern: {
type: 'Identifier',
start: 7,
end: 10,
name: 'req',
typeAnnotation: null,
optional: false
},
accessibility: null,
readonly: false,
override: false
},
{
type: 'FormalParameter',
start: 12,
end: 15,
decorators: [],
pattern: {
type: 'Identifier',
start: 12,
end: 15,
name: 'res',
typeAnnotation: null,
optional: false
},
accessibility: null,
readonly: false,
override: false
}
]
},
returnType: null,
body: {
type: 'FunctionBody',
start: 20,
end: 892,
directives: [],
statements: [
{
type: 'VariableDeclaration',
start: 26,
end: 102,
kind: 'const',
declarations: [
{
type: 'VariableDeclarator',
start: 32,
end: 68,
id: {
type: 'Identifier',
start: 32,
end: 36,
name: 'var1',
typeAnnotation: null,
optional: false
},
init: {
type: 'ComputedMemberExpression',
start: 39,
end: 68,
object: {
type: 'StaticMemberExpression',
start: 39,
end: 50,
object: {
type: 'Identifier',
start: 39,
end: 42,
name: 'req'
},
property: {
type: 'Identifier',
start: 43,
end: 50,
name: 'headers'
},
optional: false
},
expression: {
type: 'StringLiteral',
start: 51,
end: 67,
value: 'X-Telemetry-ID'
},
optional: false
},
definite: false
},
{
type: 'VariableDeclarator',
start: 70,
end: 101,
id: {
type: 'Identifier',
start: 70,
end: 74,
name: 'var2',
typeAnnotation: null,
optional: false
},
init: {
type: 'ComputedMemberExpression',
start: 77,
end: 101,
object: {
type: 'StaticMemberExpression',
start: 77,
end: 88,
object: {
type: 'Identifier',
start: 77,
end: 80,
name: 'req'
},
property: {
type: 'Identifier',
start: 81,
end: 88,
name: 'cookies'
},
optional: false
},
expression: {
type: 'StringLiteral',
start: 89,
end: 100,
value: 'X-User-ID'
},
optional: false
},
definite: false
}
],
declare: false
},
{
type: 'ExpressionStatement',
start: 107,
end: 324,
expression: {
type: 'CallExpression',
start: 107,
end: 323,
arguments: [
{
type: 'ArrowFunctionExpression',
start: 117,
end: 308,
expression: false,
async: false,
typeParameters: null,
params: {
type: 'FormalParameters',
start: 117,
end: 119,
kind: 'ArrowFormalParameters',
items: []
},
returnType: null,
body: {
type: 'FunctionBody',
start: 123,
end: 308,
directives: [],
statements: [
{
type: 'VariableDeclaration',
start: 131,
end: 142,
kind: 'let',
declarations: [
{
type: 'VariableDeclarator',
start: 135,
end: 141,
id: {
type: 'Identifier',
start: 135,
end: 141,
name: 'unload',
typeAnnotation: null,
optional: false
},
init: null,
definite: false
}
],
declare: false
},
{
type: 'ExpressionStatement',
start: 149,
end: 281,
expression: {
type: 'CallExpression',
start: 149,
end: 280,
arguments: [],
callee: {
type: 'ParenthesizedExpression',
start: 149,
end: 278,
expression: {
type: 'ArrowFunctionExpression',
start: 150,
end: 277,
expression: false,
async: true,
typeParameters: null,
params: {
type: 'FormalParameters',
start: 156,
end: 158,
kind: 'ArrowFormalParameters',
items: []
},
returnType: null,
body: {
type: 'FunctionBody',
start: 162,
end: 277,
directives: [],
statements: [
{
type: 'ExpressionStatement',
start: 172,
end: 216,
expression: {
type: 'AwaitExpression',
start: 172,
end: 215,
argument: {
type: 'CallExpression',
start: 178,
end: 215,
arguments: [
{
type: 'ComputedMemberExpression',
start: 191,
end: 214,
object: {
type: 'StaticMemberExpression',
start: 191,
end: 201,
object: {
type: 'Identifier',
start: 191,
end: 194,
name: 'req'
},
property: {
type: 'Identifier',
start: 195,
end: 201,
name: 'params'
},
optional: false
},
expression: {
type: 'StringLiteral',
start: 202,
end: 213,
value: 'x-user-id'
},
optional: false
}
],
callee: {
type: 'Identifier',
start: 178,
end: 190,
name: 'xUserConnect'
},
typeParameters: null,
optional: false
}
}
},
{
type: 'ExpressionStatement',
start: 225,
end: 269,
expression: {
type: 'ParenthesizedExpression',
start: 225,
end: 268,
expression: {
type: 'AssignmentExpression',
start: 226,
end: 267,
operator: '=',
left: {
type: 'ObjectAssignmentTarget',
start: 226,
end: 236,
properties: [
{
type: 'AssignmentTargetPropertyIdentifier',
start: 228,
end: 234,
binding: {
type: 'Identifier',
start: 228,
end: 234,
name: 'unload'
},
init: null
}
]
},
right: {
type: 'AwaitExpression',
start: 239,
end: 267,
argument: {
type: 'CallExpression',
start: 245,
end: 267,
arguments: [
{
type: 'Identifier',
start: 262,
end: 266,
name: 'var1'
}
],
callee: {
type: 'Identifier',
start: 245,
end: 261,
name: 'loadTelemetryApp'
},
typeParameters: null,
optional: false
}
}
}
}
}
]
},
scopeId: null
}
},
typeParameters: null,
optional: false
}
},
{
type: 'ReturnStatement',
start: 288,
end: 302,
argument: {
type: 'Identifier',
start: 295,
end: 301,
name: 'unload'
}
}
]
},
scopeId: null
},
{
type: 'ArrayExpression',
start: 310,
end: 322,
elements: [
{
type: 'Identifier',
start: 311,
end: 315,
name: 'var1'
},
{
type: 'Identifier',
start: 317,
end: 321,
name: 'var2'
}
]
}
],
callee: {
type: 'Identifier',
start: 107,
end: 116,
name: 'useEffect'
},
typeParameters: null,
optional: false
}
},
{
type: 'VariableDeclaration',
start: 329,
end: 621,
kind: 'const',
declarations: [
{
type: 'VariableDeclarator',
start: 335,
end: 369,
id: {
type: 'Identifier',
start: 335,
end: 341,
name: 'result',
typeAnnotation: null,
optional: false
},
init: {
type: 'AwaitExpression',
start: 344,
end: 369,
argument: {
type: 'CallExpression',
start: 350,
end: 369,
arguments: [],
callee: {
type: 'Identifier',
start: 350,
end: 367,
name: 'someExpensiveTask'
},
typeParameters: null,
optional: false
}
},
definite: false
},
{
type: 'VariableDeclarator',
start: 371,
end: 413,
id: {
type: 'Identifier',
start: 371,
end: 378,
name: 'confirm',
typeAnnotation: null,
optional: false
},
init: {
type: 'AwaitExpression',
start: 381,
end: 413,
argument: {
type: 'CallExpression',
start: 387,
end: 413,
arguments: [
{
type: 'Identifier',
start: 401,
end: 404,
name: 'req'
},
{
type: 'Identifier',
start: 406,
end: 412,
name: 'result'
}
],
callee: {
type: 'Identifier',
start: 387,
end: 400,
name: 'waitToConfirm'
},
typeParameters: null,
optional: false
}
},
definite: false
},
{
type: 'VariableDeclarator',
start: 415,
end: 432,
id: {
type: 'ObjectPattern',
start: 415,
end: 426,
properties: [
{
type: 'BindingProperty',
start: 417,
end: 424,
key: {
type: 'Identifier',
start: 417,
end: 424,
name: 'cookies'
},
value: {
type: 'Identifier',
start: 417,
end: 424,
name: 'cookies',
typeAnnotation: null,
optional: false
},
shorthand: true,
computed: false
}
],
typeAnnotation: null,
optional: false
},
init: {
type: 'Identifier',
start: 429,
end: 432,
name: 'req'
},
definite: false
},
{
type: 'VariableDeclarator',
start: 434,
end: 448,
id: {
type: 'ObjectPattern',
start: 434,
end: 442,
properties: [
{
type: 'BindingProperty',
start: 436,
end: 440,
key: {
type: 'Identifier',
start: 436,
end: 440,
name: 'send'
},
value: {
type: 'Identifier',
start: 436,
end: 440,
name: 'send',
typeAnnotation: null,
optional: false
},
shorthand: true,
computed: false
}
],
typeAnnotation: null,
optional: false
},
init: {
type: 'Identifier',
start: 445,
end: 448,
name: 'res'
},
definite: false
},
{
type: 'VariableDeclarator',
start: 450,
end: 471,
id: {
type: 'Identifier',
start: 450,
end: 457,
name: 'queries',
typeAnnotation: null,
optional: false
},
init: {
type: 'StaticMemberExpression',
start: 460,
end: 471,
object: {
type: 'Identifier',
start: 460,
end: 463,
name: 'req'
},
property: {
type: 'Identifier',
start: 464,
end: 471,
name: 'queries'
},
optional: false
},
definite: false
},
{
type: 'VariableDeclarator',
start: 473,
end: 499,
id: {
type: 'ObjectPattern',
start: 473,
end: 485,
properties: [
{
type: 'BindingProperty',
start: 475,
end: 483,
key: {
type: 'Identifier',
start: 475,
end: 483,
name: 'compress'
},
value: {
type: 'Identifier',
start: 475,
end: 483,
name: 'compress',
typeAnnotation: null,
optional: false
},
shorthand: true,
computed: false
}
],
typeAnnotation: null,
optional: false
},
init: {
type: 'StaticMemberExpression',
start: 488,
end: 499,
object: {
type: 'Identifier',
start: 488,
end: 491,
name: 'req'
},
property: {
type: 'Identifier',
start: 492,
end: 499,
name: 'queries'
},
optional: false
},
definite: false
},
{
type: 'VariableDeclarator',
start: 501,
end: 525,
id: {
type: 'ObjectPattern',
start: 501,
end: 515,
properties: [
{
type: 'BindingProperty',
start: 503,
end: 513,
key: {
type: 'Identifier',
start: 503,
end: 513,
name: 'check_zstd'
},
value: {
type: 'Identifier',
start: 503,
end: 513,
name: 'check_zstd',
typeAnnotation: null,
optional: false
},
shorthand: true,
computed: false
}
],
typeAnnotation: null,
optional: false
},
init: {
type: 'Identifier',
start: 518,
end: 525,
name: 'queries'
},
definite: false
},
{
type: 'VariableDeclarator',
start: 527,
end: 546,
id: {
type: 'Identifier',
start: 527,
end: 533,
name: 'values',
typeAnnotation: null,
optional: false
},
init: {
type: 'ArrayExpression',
start: 536,
end: 546,
elements: [
{
type: 'StaticMemberExpression',
start: 537,
end: 545,
object: {
type: 'Identifier',
start: 537,
end: 540,
name: 'req'
},
property: {
type: 'Identifier',
start: 541,
end: 545,
name: 'body'
},
optional: false
}
]
},
definite: false
},
{
type: 'VariableDeclarator',
start: 548,
end: 585,
id: {
type: 'Identifier',
start: 548,
end: 551,
name: 'foo',
typeAnnotation: null,
optional: false
},
init: {
type: 'ObjectExpression',
start: 554,
end: 585,
properties: [
{
type: 'ObjectProperty',
start: 562,
end: 579,
kind: 'init',
key: {
type: 'Identifier',
start: 562,
end: 565,
name: 'bar'
},
value: {
type: 'StaticMemberExpression',
start: 567,
end: 579,
object: {
type: 'StaticMemberExpression',
start: 567,
end: 575,
object: {
type: 'Identifier',
start: 567,
end: 570,
name: 'req'
},
property: {
type: 'Identifier',
start: 571,
end: 575,
name: 'body'
},
optional: false
},
property: {
type: 'Identifier',
start: 576,
end: 579,
name: 'bar'
},
optional: false
},
init: null,
method: false,
shorthand: false,
computed: false
}
]
},
definite: false
},
{
type: 'VariableDeclarator',
start: 587,
end: 620,
id: {
type: 'Identifier',
start: 587,
end: 595,
name: 'template',
typeAnnotation: null,
optional: false
},
init: {
type: 'TemplateLiteral',
start: 598,
end: 620,
quasis: [
{
type: 'TemplateElement',
start: 599,
end: 599,
tail: false,
value: {
raw: '',
cooked: ''
}
},
{
type: 'TemplateElement',
start: 619,
end: 619,
tail: true,
value: {
raw: '',
cooked: ''
}
}
],
expressions: [
{
type: 'StaticMemberExpression',
start: 601,
end: 618,
object: {
type: 'StaticMemberExpression',
start: 601,
end: 611,
object: {
type: 'Identifier',
start: 601,
end: 604,
name: 'req'
},
property: {
type: 'Identifier',
start: 605,
end: 611,
name: 'params'
},
optional: false
},
property: {
type: 'Identifier',
start: 612,
end: 618,
name: 'mailId'
},
optional: false
}
]
},
definite: false
}
],
declare: false
},
{
type: 'ExpressionStatement',
start: 626,
end: 655,
expression: {
type: 'AwaitExpression',
start: 626,
end: 654,
argument: {
type: 'CallExpression',
start: 632,
end: 654,
arguments: [
{
type: 'StaticMemberExpression',
start: 639,
end: 653,
object: {
type: 'Identifier',
start: 639,
end: 646,
name: 'queries'
},
property: {
type: 'Identifier',
start: 647,
end: 653,
name: 'elemId'
},
optional: false
}
],
callee: {
type: 'Identifier',
start: 632,
end: 638,
name: 'testMe'
},
typeParameters: null,
optional: false
}
}
},
{
type: 'ExpressionStatement',
start: 660,
end: 683,
expression: {
type: 'CallExpression',
start: 660,
end: 682,
arguments: [
{
type: 'StaticMemberExpression',
start: 671,
end: 681,
object: {
type: 'Identifier',
start: 671,
end: 674,
name: 'req'
},
property: {
type: 'Identifier',
start: 675,
end: 681,
name: 'params'
},
optional: false
}
],
callee: {
type: 'Identifier',
start: 660,
end: 670,
name: 'shouldSync'
},
typeParameters: null,
optional: false
}
},
{
type: 'ExpressionStatement',
start: 688,
end: 730,
expression: {
type: 'AwaitExpression',
start: 688,
end: 729,
argument: {
type: 'CallExpression',
start: 694,
end: 729,
arguments: [
{
type: 'StaticMemberExpression',
start: 704,
end: 715,
object: {
type: 'Identifier',
start: 704,
end: 707,
name: 'req'
},
property: {
type: 'Identifier',
start: 708,
end: 715,
name: 'headers'
},
optional: false
},
{
type: 'StaticMemberExpression',
start: 717,
end: 728,
object: {
type: 'Identifier',
start: 717,
end: 720,
name: 'req'
},
property: {
type: 'Identifier',
start: 721,
end: 728,
name: 'cookies'
},
optional: false
}
],
callee: {
type: 'Identifier',
start: 694,
end: 703,
name: 'checkAuth'
},
typeParameters: null,
optional: false
}
}
},
{
type: 'ExpressionStatement',
start: 735,
end: 766,
expression: {
type: 'AwaitExpression',
start: 735,
end: 765,
argument: {
type: 'CallExpression',
start: 741,
end: 765,
arguments: [
{
type: 'Identifier',
start: 749,
end: 755,
name: 'result'
},
{
type: 'Identifier',
start: 757,
end: 764,
name: 'confirm'
}
],
callee: {
type: 'Identifier',
start: 741,
end: 748,
name: 'itWorks'
},
typeParameters: null,
optional: false
}
}
},
{
type: 'ExpressionStatement',
start: 771,
end: 792,
expression: {
type: 'CallExpression',
start: 771,
end: 791,
arguments: [
{
type: 'Identifier',
start: 783,
end: 790,
name: 'confirm'
}
],
callee: {
type: 'Identifier',
start: 771,
end: 782,
name: 'itWorksSync'
},
typeParameters: null,
optional: false
}
},
{
type: 'ExpressionStatement',
start: 797,
end: 854,
expression: {
type: 'AwaitExpression',
start: 797,
end: 853,
argument: {
type: 'CallExpression',
start: 803,
end: 853,
arguments: [],
callee: {
type: 'ParenthesizedExpression',
start: 803,
end: 851,
expression: {
type: 'ArrowFunctionExpression',
start: 804,
end: 850,
expression: false,
async: true,
typeParameters: null,
params: {
type: 'FormalParameters',
start: 810,
end: 812,
kind: 'ArrowFormalParameters',
items: []
},
returnType: null,
body: {
type: 'FunctionBody',
start: 816,
end: 850,
directives: [],
statements: [
{
type: 'ExpressionStatement',
start: 824,
end: 844,
expression: {
type: 'AwaitExpression',
start: 824,
end: 843,
argument: {
type: 'CallExpression',
start: 830,
end: 843,
arguments: [],
callee: {
type: 'Identifier',
start: 830,
end: 841,
name: 'someSQLTask'
},
typeParameters: null,
optional: false
}
}
}
]
},
scopeId: null
}
},
typeParameters: null,
optional: false
}
}
},
{
type: 'ReturnStatement',
start: 859,
end: 888,
argument: {
type: 'ObjectExpression',
start: 866,
end: 887,
properties: [
{
type: 'ObjectProperty',
start: 868,
end: 885,
kind: 'init',
key: {
type: 'Identifier',
start: 868,
end: 874,
name: 'status'
},
value: {
type: 'StringLiteral',
start: 876,
end: 885,
value: 'success'
},
init: null,
method: false,
shorthand: false,
computed: false
}
]
}
}
]
},
scopeId: null
}
}
],
scopeId: null
} as const;
Copy, paste into your IDE and try result. and see how works smartly and you'll see all properties as-is (due of how as const TypeScript trick works)
Ah ha. Thanks for clarifying. We are working on infrastructure to generate TS type definitions from our internal Rust types. It'll take a little time to do that (not quite as simple as it sounds!), but it is coming.
Thank you @overlookmotel
if there need any help at JavaScript/TypeScript-side or any testing, i would like to help with that
Thanks very much for the offer. I've made a note to call on you when we have type def generation working.
@dalisoft could you please help me understand how Propose 2 can work with dynamic value?
@escaton It will work as expected because all values (even dynamic) will be as types so it gets as 1 | 2 for example instead of type number.
In order for TS to infer something with as const assertion, that something needs to be literal, which is impossible for dynamic values.
From the TS diagnostic:
A 'const' assertions can only be applied to references to enum members, or string, number, boolean, array, or object literals.
@escaton Seems i don’t understand what you means by dynamic values.
please give me example of dynamic values and i will try to use as const.
i am not pro TS developer, just average developer with TS experience
What happens when AST is parsed by Rust and then transferred to JS is conceptually not different from JSON.parse So how would you use as const to get type inference in this case?
const ast = JSON.parse('{"body": ["ImportDecalration","ExportDecalration"]}')
ast.body
// ^?
For such case I don’t know but with for my own project i am did
- parse AST
- write to JSON file
- Import JSON file via import
- Use TypeScript resolve json tsconfig parameter
Indeed, that will work, but this is not quite the mainstream usage of the ast parsers :)
I've been looking at using the node bindings for oxc-parser (did an experiment to use oxc instead of esbuild + acorn for nuxt). Because of the slight but significant differences from acorn, it would be hugely helpful to have types for the oxc AST. Let me know if there's anything I can do to help, either in implementing the type generation, or in testing the output. 🙏
@danielroe Thanks for the offer. I'm afraid that's going to be a while, because generating types is tied up with other major changes we're making to how AST data is transported from Rust to JS. And for the same reasons, it's unfortunately not something that's really accessible to contributors to work on either.
I'm really sorry that's a rubbish answer! We would dearly like to receive help from the community, but it's just a lot more complicated than it might appear. It goes deep!
When we get movement on this and have type generation working, I hope you don't mind if we take up your offer to help with testing.
#6796 alters parseSync and parseAsync to return AST as a JS object.
We should still add a json option for users who want to skip JSON.parse(). e.g. if they want to save the AST to disk as JSON.
We should still add a json option for users who want to skip JSON.parse(). e.g. if they want to save the AST to disk as JSON.
Will defer when requirement arises.
Types: https://www.npmjs.com/package/@oxc-project/types
Visitor: https://www.npmjs.com/package/estree-walker
@danielroe oxc-parser package now has types. If you'd be willing to test them out and report any problems, that'd be amazing.
@dalisoft A lot of time has passed, and very possibly your interest has moved on by now. But I said I'd let you know when we have TS type defs, so am doing so!
The AST on JS side is now fully ESTree-compatible for JS syntax, and also has types. Any testing of the types and reports if any of them are wrong would be very much appreciated.