feat(rome_json_parser): JSON Lexer

[MichaReiser]

Summary

Part of #2351

This is the first step toward our own JSON parser. The PR implements a basic JSON lexer with basic error recovery.

Non-standard features implemented by the lexer for better error recovery:

• Support for comments
• Support for identifiers
• Support for single-quoted string

This PR introduces a new rome_js_unicode_table crate that is shared between the json and JS parser and allows for fast unicode resolution.

Test Plan

I wrote a handful of manual tests and copied the JSONTestSuite

[netlify[bot]]

✅ Deploy Preview for docs-rometools canceled.

Name	Link
🔨 Latest commit	aaac99a1f5b61db608c68ea54aa6ac15a2190e13
🔍 Latest deploy log	https://app.netlify.com/sites/docs-rometools/deploys/637dd782bb58cc0008343fdf

[github-actions[bot]]

Parser conformance results on ubuntu-latest

js/262

Test result	main count	This PR count	Difference
Total	45879	45879	0
Passed	44936	44936	0
Failed	943	943	0
Panics	0	0	0
Coverage	97.94%	97.94%	0.00%

jsx/babel

Test result	main count	This PR count	Difference
Total	39	39	0
Passed	36	36	0
Failed	3	3	0
Panics	0	0	0
Coverage	92.31%	92.31%	0.00%

symbols/microsoft

Test result	main count	This PR count	Difference
Total	5946	5946	0
Passed	1757	1757	0
Failed	4189	4189	0
Panics	0	0	0
Coverage	29.55%	29.55%	0.00%

ts/babel

Test result	main count	This PR count	Difference
Total	588	588	0
Passed	519	519	0
Failed	69	69	0
Panics	0	0	0
Coverage	88.27%	88.27%	0.00%

ts/microsoft

Test result	main count	This PR count	Difference
Total	16257	16257	0
Passed	12397	12397	0
Failed	3860	3860	0
Panics	0	0	0
Coverage	76.26%	76.26%	0.00%

[MichaReiser]

!bench_parser

[MichaReiser]

!bench_parser

[calibre-analytics[bot]]

Comparing feat(rome_json_parser): JSON Lexer Snapshot #5 to median since last deploy of rome.tools.

	LCP?	CLS?	TBT?
Overall Median across all pages and test profiles	2.25s  from 264ms	0.0 no change	136ms no change
Chrome Desktop Chrome Desktop • Cable	2.25s  from 264ms	0.0 no change	328ms  from 22ms
iPhone, 4G LTE iPhone 12 • 4G LTE	1.08s  from 239ms	0.0 no change	11ms no change
Motorola Moto G Power, 3G connection Motorola Moto G Power • Regular 3G	16.4s  from 1.06s	0.0 no change	136ms no change

1 page tested

 Home

Browser previews

Chrome Desktop	iPhone, 4G LTE	Motorola Moto G Power, 3G connection

Most significant changes

	Value	Budget
Total JavaScript Size in Bytes Chrome Desktop	5.35 MB  from 86.8 KB
Total JavaScript Size in Bytes iPhone, 4G LTE	5.35 MB  from 86.8 KB
Total JavaScript Size in Bytes Motorola Moto G Power, 3G connection	5.35 MB  from 86.8 KB
JS Parse & Compile Motorola Moto G Power, 3G connection	1.66s  from 27ms
JS Parse & Compile iPhone, 4G LTE	467ms  from 12ms

_{27 other significant changes: JS Parse & Compile on Chrome Desktop, Largest Contentful Paint on Motorola Moto G Power, 3G connection, First Contentful Paint on Motorola Moto G Power, 3G connection, Total Blocking Time on Chrome Desktop, Total CSS Size in Bytes on Chrome Desktop, Total CSS Size in Bytes on iPhone, 4G LTE, Total CSS Size in Bytes on Motorola Moto G Power, 3G connection, Time to Interactive on Motorola Moto G Power, 3G connection, Total Page Size in Bytes on Chrome Desktop, Total Page Size in Bytes on iPhone, 4G LTE, Total Page Size in Bytes on Motorola Moto G Power, 3G connection, Time to Interactive on Chrome Desktop, Number of Requests on Motorola Moto G Power, 3G connection, Number of Requests on Chrome Desktop, Number of Requests on iPhone, 4G LTE, First Contentful Paint on Chrome Desktop, Largest Contentful Paint on Chrome Desktop, Speed Index on Motorola Moto G Power, 3G connection, Time to Interactive on iPhone, 4G LTE, Largest Contentful Paint on iPhone, 4G LTE, First Contentful Paint on iPhone, 4G LTE, Speed Index on Chrome Desktop, Total HTML Size in Bytes on Chrome Desktop, Total HTML Size in Bytes on iPhone, 4G LTE, Total HTML Size in Bytes on Motorola Moto G Power, 3G connection, Lighthouse Performance Score on Motorola Moto G Power, 3G connection, Lighthouse Performance Score on Chrome Desktop}

_{Calibre: Site dashboard | View this PR | Edit settings | View documentation}

[github-actions[bot]]

Parser Benchmark Results

group                                 main                                   pr
-----                                 ----                                   --
parser/checker.ts                     1.00    149.8±9.26ms    17.4 MB/sec    1.02   152.9±12.21ms    17.0 MB/sec
parser/compiler.js                    1.01     90.7±5.30ms    11.6 MB/sec    1.00     89.7±5.89ms    11.7 MB/sec
parser/d3.min.js                      1.00     49.7±3.38ms     5.3 MB/sec    1.11     55.3±4.83ms     4.7 MB/sec
parser/dojo.js                        1.02      4.3±0.19ms    16.0 MB/sec    1.00      4.2±0.23ms    16.3 MB/sec
parser/ios.d.ts                       1.01    129.1±7.71ms    14.5 MB/sec    1.00    128.3±7.33ms    14.5 MB/sec
parser/jquery.min.js                  1.00     14.3±1.45ms     5.8 MB/sec    1.02     14.6±1.03ms     5.7 MB/sec
parser/math.js                        1.05    113.1±4.20ms     5.7 MB/sec    1.00    107.7±5.24ms     6.0 MB/sec
parser/parser.ts                      1.00      3.1±0.19ms    15.6 MB/sec    1.03      3.2±0.15ms    15.1 MB/sec
parser/pixi.min.js                    1.00     69.4±3.66ms     6.3 MB/sec    1.02     71.1±4.60ms     6.2 MB/sec
parser/react-dom.production.min.js    1.00     17.9±1.14ms     6.4 MB/sec    1.08     19.3±1.06ms     6.0 MB/sec
parser/react.production.min.js        1.00   929.1±44.13µs     6.6 MB/sec    1.02   950.4±41.41µs     6.5 MB/sec
parser/router.ts                      1.00      2.5±0.12ms    25.4 MB/sec    1.07      2.7±0.29ms    23.7 MB/sec
parser/tex-chtml-full.js              1.09    151.9±5.76ms     6.0 MB/sec    1.00    139.6±7.88ms     6.5 MB/sec
parser/three.min.js                   1.04     76.9±3.85ms     7.6 MB/sec    1.00     73.7±3.90ms     8.0 MB/sec
parser/typescript.js                  1.01   595.9±35.82ms    15.9 MB/sec    1.00   591.4±29.71ms    16.1 MB/sec
parser/vue.global.prod.js             1.03     23.8±2.56ms     5.1 MB/sec    1.00     23.0±1.92ms     5.2 MB/sec

[Conaclos]

Non-standard features implemented by the lexer

Worth to note that extensions of JSON exist:

• JSONC
• JSON5
• JSON6

Rome could support the most common: JSON and JSONC