TypeScript

TypeScript copied to clipboard

Reame
Issues

Remove reportErrors check in relateVariances

Open jakebailey opened this issue 2 years ago • 74 comments

Fixes #55217 Fixes #61524

As noted by @Andarist, this check is what causes things to differ between errors and non-error modes in relation, causing oddities when the same relation is observed in both ways but in different orders.

No doubt this is a bad idea but I wanted to run the test suite to see what happens.

Bunch of WIP code is left in the PR, I'll remove/refactor it if this code actually looks acceptable.

Jul 31 '23 23:07 jakebailey

@typescript-bot test this @typescript-bot test top100 @typescript-bot user test this @typescript-bot user test tsserver @typescript-bot test tsserver top100 @typescript-bot run dt @typescript-bot perf test this @typescript-bot pack this

Jul 31 '23 23:07 jakebailey

Heya @jakebailey, I've started to run the diff-based user code test suite on this PR at df695d9adb838785a10e1c354596c0e39da71847. You can monitor the build here.

Update: The results are in!

Jul 31 '23 23:07 typescript-bot

Heya @jakebailey, I've started to run the diff-based user code test suite (tsserver) on this PR at df695d9adb838785a10e1c354596c0e39da71847. You can monitor the build here.

Update: The results are in!

Jul 31 '23 23:07 typescript-bot

Heya @jakebailey, I've started to run the tarball bundle task on this PR at df695d9adb838785a10e1c354596c0e39da71847. You can monitor the build here.

Jul 31 '23 23:07 typescript-bot

Heya @jakebailey, I've started to run the extended test suite on this PR at df695d9adb838785a10e1c354596c0e39da71847. You can monitor the build here.

Jul 31 '23 23:07 typescript-bot

Heya @jakebailey, I've started to run the parallelized Definitely Typed test suite on this PR at df695d9adb838785a10e1c354596c0e39da71847. You can monitor the build here.

Update: The results are in!

Jul 31 '23 23:07 typescript-bot

Heya @jakebailey, I've started to run the perf test suite on this PR at df695d9adb838785a10e1c354596c0e39da71847. You can monitor the build here.

Update: The results are in!

Jul 31 '23 23:07 typescript-bot

Heya @jakebailey, I've started to run the diff-based top-repos suite on this PR at df695d9adb838785a10e1c354596c0e39da71847. You can monitor the build here.

Update: The results are in!

Jul 31 '23 23:07 typescript-bot

Heya @jakebailey, I've started to run the diff-based top-repos suite (tsserver) on this PR at df695d9adb838785a10e1c354596c0e39da71847. You can monitor the build here.

Update: The results are in!

Jul 31 '23 23:07 typescript-bot

Hey @jakebailey, I've packed this into an installable tgz. You can install it for testing by referencing it in your package.json like so:

{
    "devDependencies": {
        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/156090/artifacts?artifactName=tgz&fileId=4E1314C8F2DBC21591658BB650878A546D6F6E3C21856E1045D90F437FA08F0902&fileName=/typescript-5.2.0-insiders.20230731.tgz"
    }
}

and then running npm install.

There is also a playground for this build and an npm module you can use via "typescript": "npm:@typescript-deploys/[email protected]".;

Jul 31 '23 23:07 typescript-bot

@jakebailey Here are the results of running the user test suite comparing main and refs/pull/55222/merge:

Everything looks good!

Aug 01 '23 00:08 typescript-bot

@jakebailey Here are the results of running the user test suite comparing main and refs/pull/55222/merge:

There were infrastructure failures potentially unrelated to your change:

1 instance of "Unknown failure"
1 instance of "Package install failed"

Otherwise...

Something interesting changed - please have a look.

Details

rxjs-src

/mnt/ts_downloads/rxjs-src/build.sh

[NEW] error TS2428: All declarations of 'WeakMap' must have identical type parameters.
- /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.collection.d.ts(63,11)
- /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.iterable.d.ts(162,11)
- /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.symbol.wellknown.d.ts(140,11)
- /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.collection.d.ts(63,11)
- /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.iterable.d.ts(162,11)
- /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.symbol.wellknown.d.ts(140,11)
- /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.collection.d.ts(63,11)
- /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.iterable.d.ts(162,11)
- /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.symbol.wellknown.d.ts(140,11)
- /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.collection.d.ts(63,11)
- /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.iterable.d.ts(162,11)
- /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.symbol.wellknown.d.ts(140,11)
- /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.collection.d.ts(63,11)
- /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.iterable.d.ts(162,11)
- /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.symbol.wellknown.d.ts(140,11)
[MISSING] error TS2428: All declarations of 'WeakMap' must have identical type parameters.
- /home/vsts/work/1/s/typescript-main/lib/lib.es2015.collection.d.ts(63,11)
- /home/vsts/work/1/s/typescript-main/lib/lib.es2015.iterable.d.ts(162,11)
- /home/vsts/work/1/s/typescript-main/lib/lib.es2015.symbol.wellknown.d.ts(140,11)
- /home/vsts/work/1/s/typescript-main/lib/lib.es2015.collection.d.ts(63,11)
- /home/vsts/work/1/s/typescript-main/lib/lib.es2015.iterable.d.ts(162,11)
- /home/vsts/work/1/s/typescript-main/lib/lib.es2015.symbol.wellknown.d.ts(140,11)
- /home/vsts/work/1/s/typescript-main/lib/lib.es2015.collection.d.ts(63,11)
- /home/vsts/work/1/s/typescript-main/lib/lib.es2015.iterable.d.ts(162,11)
- /home/vsts/work/1/s/typescript-main/lib/lib.es2015.symbol.wellknown.d.ts(140,11)
- /home/vsts/work/1/s/typescript-main/lib/lib.es2015.collection.d.ts(63,11)
- /home/vsts/work/1/s/typescript-main/lib/lib.es2015.iterable.d.ts(162,11)
- /home/vsts/work/1/s/typescript-main/lib/lib.es2015.symbol.wellknown.d.ts(140,11)
- /home/vsts/work/1/s/typescript-main/lib/lib.es2015.collection.d.ts(63,11)
- /home/vsts/work/1/s/typescript-main/lib/lib.es2015.iterable.d.ts(162,11)
- /home/vsts/work/1/s/typescript-main/lib/lib.es2015.symbol.wellknown.d.ts(140,11)

Aug 01 '23 00:08 typescript-bot

@jakebailey The results of the perf run you requested are in!

Here they are:

Compiler

Comparison Report - main..55222

_Metric	_main	₅₅₂₂₂	_Delta	_Best	_Worst	_p-value
_{Angular - node (v18.10.0, x64)}
_{Memory used}	_{368,785k (± 0.01%)}	_{368,794k (± 0.01%)}	_~	_368,756k	_368,821k	_{p=0.689 n=6}
_{Parse Time}	_{3.40s (± 0.55%)}	_{3.40s (± 0.42%)}	_~	_3.38s	_3.42s	_{p=0.685 n=6}
_{Bind Time}	_{1.12s (± 0.92%)}	_{1.12s (± 0.73%)}	_~	_1.11s	_1.13s	_{p=0.932 n=6}
_{Check Time}	_{8.95s (± 0.34%)}	_{9.00s (± 0.44%)}	_{+0.05s (+ 0.60%)}	_8.93s	_9.04s	_{p=0.045 n=6}
_{Emit Time}	_{7.57s (± 1.03%)}	_{7.58s (± 0.76%)}	_~	_7.50s	_7.65s	_{p=1.000 n=6}
_{Total Time}	_{21.05s (± 0.38%)}	_{21.10s (± 0.49%)}	_~	_20.95s	_21.22s	_{p=0.421 n=6}
_{Compiler-Unions - node (v18.10.0, x64)}
_{Memory used}	_{193,051k (± 1.18%)}	_{194,922k (± 1.56%)}	_~	_192,120k	_197,736k	_{p=0.128 n=6}
_{Parse Time}	_{1.51s (± 0.65%)}	_{1.50s (± 0.98%)}	_~	_1.48s	_1.52s	_{p=0.247 n=6}
_{Bind Time}	_{0.78s (± 0.66%)}	_{0.77s (± 0.67%)}	_~	_0.77s	_0.78s	_{p=0.311 n=6}
_{Check Time}	_{9.47s (± 0.42%)}	_{9.46s (± 0.30%)}	_~	_9.42s	_9.50s	_{p=1.000 n=6}
_{Emit Time}	_{2.76s (± 0.38%)}	_{2.72s (± 1.02%)}	_~	_2.70s	_2.77s	_{p=0.052 n=6}
_{Total Time}	_{14.51s (± 0.35%)}	_{14.46s (± 0.18%)}	_~	_14.42s	_14.48s	_{p=0.053 n=6}
_{Monaco - node (v18.10.0, x64)}
_{Memory used}	_{347,755k (± 0.01%)}	_{347,772k (± 0.01%)}	_~	_347,741k	_347,813k	_{p=0.471 n=6}
_{Parse Time}	_{2.62s (± 1.05%)}	_{2.63s (± 1.19%)}	_~	_2.58s	_2.66s	_{p=0.418 n=6}
_{Bind Time}	_{1.01s (± 0.74%)}	_{1.01s (± 0.51%)}	_~	_1.01s	_1.02s	_{p=0.784 n=6}
_{Check Time}	_{7.29s (± 0.38%)}	_{7.29s (± 0.64%)}	_~	_7.21s	_7.35s	_{p=0.936 n=6}
_{Emit Time}	_{4.24s (± 0.90%)}	_{4.27s (± 0.88%)}	_~	_4.22s	_4.33s	_{p=0.260 n=6}
_{Total Time}	_{15.16s (± 0.51%)}	_{15.21s (± 0.65%)}	_~	_15.03s	_15.31s	_{p=0.261 n=6}
_{TFS - node (v18.10.0, x64)}
_{Memory used}	_{301,794k (± 0.01%)}	_{301,786k (± 0.01%)}	_~	_301,751k	_301,818k	_{p=0.574 n=6}
_{Parse Time}	_{2.08s (± 1.23%)}	_{2.07s (± 1.77%)}	_~	_2.04s	_2.13s	_{p=0.871 n=6}
_{Bind Time}	_{1.13s (± 1.03%)}	_{1.12s (± 0.36%)}	_{-0.01s (- 1.18%)}	_1.11s	_1.12s	_{p=0.025 n=6}
_{Check Time}	_{6.67s (± 0.53%)}	_{6.66s (± 0.50%)}	_~	_6.61s	_6.70s	_{p=0.518 n=6}
_{Emit Time}	_{3.86s (± 0.95%)}	_{3.89s (± 0.48%)}	_~	_3.86s	_3.91s	_{p=0.259 n=6}
_{Total Time}	_{13.75s (± 0.50%)}	_{13.74s (± 0.44%)}	_~	_13.65s	_13.84s	_{p=1.000 n=6}
_{material-ui - node (v18.10.0, x64)}
_{Memory used}	_{482,550k (± 0.01%)}	_{482,585k (± 0.02%)}	_~	_482,533k	_482,774k	_{p=0.230 n=6}
_{Parse Time}	_{3.11s (± 1.18%)}	_{3.09s (± 2.29%)}	_~	_2.95s	_3.14s	_{p=0.933 n=6}
_{Bind Time}	_{0.92s (± 2.44%)}	_{0.94s (± 5.00%)}	_~	_0.91s	_1.03s	_{p=1.000 n=6}
_{Check Time}	_{17.32s (± 0.73%)}	_{17.33s (± 0.78%)}	_~	_17.09s	_17.49s	_{p=0.470 n=6}
_{Emit Time}	_{0.00s (± 0.00%)}	_{0.00s (± 0.00%)}	_~	_0.00s	_0.00s	_{p=1.000 n=6}
_{Total Time}	_{21.36s (± 0.54%)}	_{21.37s (± 0.63%)}	_~	_21.14s	_21.55s	_{p=0.334 n=6}
_{xstate - node (v18.10.0, x64)}
_{Memory used}	_{563,842k (± 0.02%)}	_{565,624k (± 0.01%)}	_{+1,782k (+ 0.32%)}	_565,514k	_565,730k	_{p=0.005 n=6}
_{Parse Time}	_{3.85s (± 0.61%)}	_{3.85s (± 0.35%)}	_~	_3.83s	_3.87s	_{p=0.806 n=6}
_{Bind Time}	_{1.64s (± 1.24%)}	_{1.64s (± 0.63%)}	_~	_1.62s	_1.65s	_{p=0.935 n=6}
_{Check Time}	_{2.81s (± 1.00%)}	_{3.03s (± 0.61%)}	_{+0.22s (+ 7.64%)}	_3.01s	_3.05s	_{p=0.005 n=6}
_{Emit Time}	_{0.08s (± 0.00%)}	_{0.08s (± 6.44%)}	_~	_0.08s	_0.09s	_{p=0.071 n=6}
_{Total Time}	_{8.39s (± 0.27%)}	_{8.61s (± 0.24%)}	_{+0.22s (+ 2.64%)}	_8.58s	_8.64s	_{p=0.005 n=6}
_{Angular - node (v16.17.1, x64)}
_{Memory used}	_{368,218k (± 0.00%)}	_{368,222k (± 0.01%)}	_~	_368,193k	_368,246k	_{p=0.575 n=6}
_{Parse Time}	_{3.57s (± 0.39%)}	_{3.55s (± 0.18%)}	_~	_3.54s	_3.56s	_{p=0.059 n=6}
_{Bind Time}	_{1.19s (± 0.63%)}	_{1.19s (± 0.43%)}	_~	_1.18s	_1.19s	_{p=0.784 n=6}
_{Check Time}	_{9.76s (± 0.72%)}	_{9.77s (± 0.54%)}	_~	_9.68s	_9.83s	_{p=1.000 n=6}
_{Emit Time}	_{8.04s (± 0.80%)}	_{8.01s (± 0.39%)}	_~	_7.96s	_8.04s	_{p=0.228 n=6}
_{Total Time}	_{22.55s (± 0.58%)}	_{22.51s (± 0.31%)}	_~	_22.37s	_22.56s	_{p=0.374 n=6}
_{Compiler-Unions - node (v16.17.1, x64)}
_{Memory used}	_{193,823k (± 0.02%)}	_{194,372k (± 0.67%)}	_~	_193,796k	_197,022k	_{p=0.261 n=6}
_{Parse Time}	_{1.59s (± 0.96%)}	_{1.59s (± 0.62%)}	_~	_1.58s	_1.61s	_{p=0.557 n=6}
_{Bind Time}	_{0.82s (± 0.92%)}	_{0.82s (± 0.50%)}	_~	_0.82s	_0.83s	_{p=1.000 n=6}
_{Check Time}	_{10.16s (± 0.48%)}	_{10.17s (± 0.58%)}	_~	_10.09s	_10.25s	_{p=0.810 n=6}
_{Emit Time}	_{2.98s (± 0.87%)}	_{3.01s (± 0.91%)}	_~	_2.97s	_3.05s	_{p=0.106 n=6}
_{Total Time}	_{15.56s (± 0.35%)}	_{15.60s (± 0.24%)}	_~	_15.55s	_15.64s	_{p=0.149 n=6}
_{Monaco - node (v16.17.1, x64)}
_{Memory used}	_{347,078k (± 0.01%)}	_{347,085k (± 0.01%)}	_~	_347,028k	_347,116k	_{p=0.630 n=6}
_{Parse Time}	_{2.77s (± 0.44%)}	_{2.76s (± 0.37%)}	_~	_2.75s	_2.78s	_{p=0.675 n=6}
_{Bind Time}	_{1.08s (± 0.48%)}	_{1.08s (± 0.70%)}	_~	_1.07s	_1.09s	_{p=0.784 n=6}
_{Check Time}	_{8.04s (± 0.61%)}	_{8.06s (± 0.33%)}	_~	_8.03s	_8.10s	_{p=0.295 n=6}
_{Emit Time}	_{4.48s (± 0.56%)}	_{4.47s (± 0.60%)}	_~	_4.44s	_4.51s	_{p=0.227 n=6}
_{Total Time}	_{16.36s (± 0.36%)}	_{16.37s (± 0.38%)}	_~	_16.30s	_16.48s	_{p=1.000 n=6}
_{TFS - node (v16.17.1, x64)}
_{Memory used}	_{301,118k (± 0.01%)}	_{301,109k (± 0.01%)}	_~	_301,082k	_301,135k	_{p=0.377 n=6}
_{Parse Time}	_{2.20s (± 0.38%)}	_{2.19s (± 0.24%)}	_~	_2.18s	_2.19s	_{p=0.073 n=6}
_{Bind Time}	_{1.22s (± 1.70%)}	_{1.21s (± 1.60%)}	_~	_1.18s	_1.23s	_{p=0.625 n=6}
_{Check Time}	_{7.34s (± 0.31%)}	_{7.33s (± 0.71%)}	_~	_7.24s	_7.38s	_{p=1.000 n=6}
_{Emit Time}	_{4.30s (± 1.17%)}	_{4.33s (± 0.97%)}	_~	_4.29s	_4.39s	_{p=0.146 n=6}
_{Total Time}	_{15.06s (± 0.25%)}	_{15.06s (± 0.57%)}	_~	_14.95s	_15.17s	_{p=0.748 n=6}
_{material-ui - node (v16.17.1, x64)}
_{Memory used}	_{481,882k (± 0.01%)}	_{481,860k (± 0.01%)}	_~	_481,811k	_481,954k	_{p=0.297 n=6}
_{Parse Time}	_{3.26s (± 0.72%)}	_{3.23s (± 0.26%)}	_~	_3.23s	_3.25s	_{p=0.122 n=6}
_{Bind Time}	_{0.96s (± 0.85%)}	_{0.95s (± 0.57%)}	_~	_0.95s	_0.96s	_{p=0.859 n=6}
_{Check Time}	_{18.34s (± 0.39%)}	_{18.34s (± 0.74%)}	_~	_18.21s	_18.57s	_{p=0.687 n=6}
_{Emit Time}	_{0.00s (± 0.00%)}	_{0.00s (± 0.00%)}	_~	_0.00s	_0.00s	_{p=1.000 n=6}
_{Total Time}	_{22.55s (± 0.36%)}	_{22.53s (± 0.63%)}	_~	_22.41s	_22.78s	_{p=0.422 n=6}
_{xstate - node (v16.17.1, x64)}
_{Memory used}	_{561,343k (± 0.01%)}	_{563,061k (± 0.01%)}	_{+1,718k (+ 0.31%)}	_562,967k	_563,164k	_{p=0.005 n=6}
_{Parse Time}	_{4.02s (± 0.30%)}	_{4.02s (± 0.43%)}	_~	_4.00s	_4.05s	_{p=0.871 n=6}
_{Bind Time}	_{1.66s (± 5.88%)}	_{1.77s (± 1.62%)}	_{+0.11s (+ 6.82%)}	_1.73s	_1.81s	_{p=0.024 n=6}
_{Check Time}	_{3.09s (± 1.60%)}	_{3.28s (± 1.26%)}	_{+0.19s (+ 6.09%)}	_3.23s	_3.34s	_{p=0.005 n=6}
_{Emit Time}	_{0.09s (± 5.53%)}	_{0.09s (± 0.00%)}	_~	_0.09s	_0.09s	_{p=0.174 n=6}
_{Total Time}	_{8.87s (± 0.58%)}	_{9.17s (± 0.82%)}	_{+0.29s (+ 3.33%)}	_9.06s	_9.28s	_{p=0.005 n=6}
_{Angular - node (v14.21.3, x64)}
_{Memory used}	_{362,126k (± 0.01%)}	_{362,093k (± 0.01%)}	_~	_362,054k	_362,155k	_{p=0.128 n=6}
_{Parse Time}	_{3.71s (± 0.14%)}	_{3.73s (± 0.37%)}	_{+0.02s (+ 0.54%)}	_3.71s	_3.74s	_{p=0.020 n=6}
_{Bind Time}	_{1.22s (± 0.00%)}	_{1.22s (± 0.62%)}	_~	_1.21s	_1.23s	_{p=0.598 n=6}
_{Check Time}	_{10.13s (± 0.37%)}	_{10.15s (± 0.56%)}	_~	_10.06s	_10.22s	_{p=0.377 n=6}
_{Emit Time}	_{8.35s (± 0.56%)}	_{8.36s (± 0.55%)}	_~	_8.30s	_8.41s	_{p=0.748 n=6}
_{Total Time}	_{23.41s (± 0.27%)}	_{23.46s (± 0.32%)}	_~	_23.33s	_23.53s	_{p=0.336 n=6}
_{Compiler-Unions - node (v14.21.3, x64)}
_{Memory used}	_{189,118k (± 0.01%)}	_{189,094k (± 0.01%)}	_~	_189,060k	_189,128k	_{p=0.092 n=6}
_{Parse Time}	_{1.62s (± 0.72%)}	_{1.61s (± 0.32%)}	_~	_1.61s	_1.62s	_{p=0.191 n=6}
_{Bind Time}	_{0.85s (± 0.61%)}	_{0.85s (± 0.61%)}	_~	_0.84s	_0.85s	_{p=1.000 n=6}
_{Check Time}	_{10.32s (± 0.70%)}	_{10.33s (± 0.44%)}	_~	_10.28s	_10.39s	_{p=0.574 n=6}
_{Emit Time}	_{3.11s (± 0.55%)}	_{3.12s (± 0.33%)}	_~	_3.10s	_3.13s	_{p=0.741 n=6}
_{Total Time}	_{15.90s (± 0.49%)}	_{15.91s (± 0.21%)}	_~	_15.86s	_15.95s	_{p=0.520 n=6}
_{Monaco - node (v14.21.3, x64)}
_{Memory used}	_{342,072k (± 0.01%)}	_{342,077k (± 0.01%)}	_~	_342,045k	_342,108k	_{p=0.936 n=6}
_{Parse Time}	_{2.80s (± 0.87%)}	_{2.80s (± 0.76%)}	_~	_2.78s	_2.84s	_{p=0.684 n=6}
_{Bind Time}	_{1.09s (± 0.50%)}	_{1.09s (± 0.37%)}	_~	_1.09s	_1.10s	_{p=0.282 n=6}
_{Check Time}	_{8.30s (± 0.26%)}	_{8.31s (± 0.43%)}	_~	_8.26s	_8.36s	_{p=0.687 n=6}
_{Emit Time}	_{4.67s (± 0.87%)}	_{4.66s (± 0.57%)}	_~	_4.63s	_4.69s	_{p=0.570 n=6}
_{Total Time}	_{16.86s (± 0.40%)}	_{16.87s (± 0.23%)}	_~	_16.81s	_16.91s	_{p=0.809 n=6}
_{TFS - node (v14.21.3, x64)}
_{Memory used}	_{296,199k (± 0.01%)}	_{296,199k (± 0.01%)}	_~	_296,181k	_296,222k	_{p=0.936 n=6}
_{Parse Time}	_{2.42s (± 0.43%)}	_{2.42s (± 0.80%)}	_~	_2.39s	_2.44s	_{p=1.000 n=6}
_{Bind Time}	_{1.08s (± 0.91%)}	_{1.08s (± 0.59%)}	_~	_1.07s	_1.09s	_{p=0.733 n=6}
_{Check Time}	_{7.66s (± 0.42%)}	_{7.67s (± 0.48%)}	_~	_7.61s	_7.70s	_{p=0.413 n=6}
_{Emit Time}	_{4.27s (± 0.69%)}	_{4.28s (± 0.84%)}	_~	_4.23s	_4.32s	_{p=0.871 n=6}
_{Total Time}	_{15.44s (± 0.22%)}	_{15.46s (± 0.40%)}	_~	_15.38s	_15.54s	_{p=0.629 n=6}
_{material-ui - node (v14.21.3, x64)}
_{Memory used}	_{477,350k (± 0.00%)}	_{477,350k (± 0.00%)}	_~	_477,319k	_477,365k	_{p=0.748 n=6}
_{Parse Time}	_{3.31s (± 0.31%)}	_{3.29s (± 0.31%)}	_~	_3.28s	_3.31s	_{p=0.067 n=6}
_{Bind Time}	_{0.99s (± 0.41%)}	_{0.99s (± 0.76%)}	_~	_0.98s	_1.00s	_{p=0.389 n=6}
_{Check Time}	_{19.15s (± 0.40%)}	_{19.21s (± 0.56%)}	_~	_19.08s	_19.34s	_{p=0.471 n=6}
_{Emit Time}	_{0.00s (± 0.00%)}	_{0.00s (± 0.00%)}	_~	_0.00s	_0.00s	_{p=1.000 n=6}
_{Total Time}	_{23.45s (± 0.30%)}	_{23.49s (± 0.48%)}	_~	_23.37s	_23.64s	_{p=0.630 n=6}
_{xstate - node (v14.21.3, x64)}
_{Memory used}	_{550,168k (± 0.00%)}	_{551,869k (± 0.01%)}	_{+1,701k (+ 0.31%)}	_551,833k	_551,905k	_{p=0.005 n=6}
_{Parse Time}	_{4.22s (± 0.61%)}	_{4.20s (± 0.51%)}	_~	_4.18s	_4.24s	_{p=0.258 n=6}
_{Bind Time}	_{1.69s (± 0.30%)}	_{1.68s (± 1.74%)}	_~	_1.62s	_1.70s	_{p=0.242 n=6}
_{Check Time}	_{3.13s (± 0.48%)}	_{3.33s (± 0.84%)}	_{+0.21s (+ 6.56%)}	_3.29s	_3.37s	_{p=0.005 n=6}
_{Emit Time}	_{0.10s (± 5.76%)}	_{0.09s (± 0.00%)}	_~	_0.09s	_0.09s	_{p=0.071 n=6}
_{Total Time}	_{9.13s (± 0.35%)}	_{9.31s (± 0.62%)}	_{+0.18s (+ 1.97%)}	_9.20s	_9.36s	_{p=0.005 n=6}

System

_{Machine Name}	_ts-ci-ubuntu
_Platform	_{linux 5.4.0-148-generic}
_Architecture	_x64
_{Available Memory}	_{16 GB}
_{Available Memory}	_{15 GB}
_CPUs	_{4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz}

Hosts

node (v18.10.0, x64)
node (v16.17.1, x64)
node (v14.21.3, x64)

Scenarios

Angular - node (v18.10.0, x64)
Angular - node (v16.17.1, x64)
Angular - node (v14.21.3, x64)
Compiler-Unions - node (v18.10.0, x64)
Compiler-Unions - node (v16.17.1, x64)
Compiler-Unions - node (v14.21.3, x64)
Monaco - node (v18.10.0, x64)
Monaco - node (v16.17.1, x64)
Monaco - node (v14.21.3, x64)
TFS - node (v18.10.0, x64)
TFS - node (v16.17.1, x64)
TFS - node (v14.21.3, x64)
material-ui - node (v18.10.0, x64)
material-ui - node (v16.17.1, x64)
material-ui - node (v14.21.3, x64)
xstate - node (v18.10.0, x64)
xstate - node (v16.17.1, x64)
xstate - node (v14.21.3, x64)

_Benchmark	_Name	_Iterations
_Current	₅₅₂₂₂	₆
_Baseline	_main	₆

TSServer

Comparison Report - main..55222

_Metric	_main	₅₅₂₂₂	_Delta	_Best	_Worst	_p-value
_{Compiler-UnionsTSServer - node (v18.10.0, x64)}
_{Req 1 - updateOpen}	_{2,525ms (± 1.15%)}	_{2,537ms (± 0.76%)}	_~	_2,511ms	_2,559ms	_{p=0.630 n=6}
_{Req 2 - geterr}	_{5,384ms (± 0.47%)}	_{5,403ms (± 0.56%)}	_~	_5,365ms	_5,435ms	_{p=0.336 n=6}
_{Req 3 - references}	_{342ms (± 0.86%)}	_{340ms (± 1.19%)}	_~	_335ms	_345ms	_{p=0.226 n=6}
_{Req 4 - navto}	_{289ms (± 0.34%)}	_{291ms (± 0.95%)}	_~	_287ms	_295ms	_{p=0.141 n=6}
_{Req 5 - completionInfo count}	_{1,356 (± 0.00%)}	_{1,356 (± 0.00%)}	_~	_1,356	_1,356	_{p=1.000 n=6}
_{Req 5 - completionInfo}	_{76ms (± 1.08%)}	_{76ms (± 3.99%)}	_~	_74ms	_82ms	_{p=0.451 n=6}
_{CompilerTSServer - node (v18.10.0, x64)}
_{Req 1 - updateOpen}	_{2,618ms (± 0.49%)}	_{2,636ms (± 1.01%)}	_~	_2,604ms	_2,670ms	_{p=0.336 n=6}
_{Req 2 - geterr}	_{4,136ms (± 0.41%)}	_{4,153ms (± 0.50%)}	_~	_4,118ms	_4,177ms	_{p=0.092 n=6}
_{Req 3 - references}	_{350ms (± 1.01%)}	_{351ms (± 0.36%)}	_~	_349ms	_353ms	_{p=0.141 n=6}
_{Req 4 - navto}	_{289ms (± 0.36%)}	_{289ms (± 0.48%)}	_~	_286ms	_290ms	_{p=0.867 n=6}
_{Req 5 - completionInfo count}	_{1,518 (± 0.00%)}	_{1,518 (± 0.00%)}	_~	_1,518	_1,518	_{p=1.000 n=6}
_{Req 5 - completionInfo}	_{79ms (± 6.58%)}	_{78ms (± 1.32%)}	_~	_77ms	_80ms	_{p=1.000 n=6}
_{xstateTSServer - node (v18.10.0, x64)}
_{Req 1 - updateOpen}	_{3,088ms (± 0.53%)}	_{3,081ms (± 0.52%)}	_~	_3,066ms	_3,107ms	_{p=0.520 n=6}
_{Req 2 - geterr}	_{1,602ms (± 0.57%)}	_{1,598ms (± 0.86%)}	_~	_1,579ms	_1,614ms	_{p=0.748 n=6}
_{Req 3 - references}	_{114ms (± 1.32%)}	_{115ms (± 2.01%)}	_~	_112ms	_118ms	_{p=0.217 n=6}
_{Req 4 - navto}	_{369ms (± 0.37%)}	_{368ms (± 0.34%)}	_~	_366ms	_369ms	_{p=0.096 n=6}
_{Req 5 - completionInfo count}	_{2,872 (± 0.00%)}	_{2,872 (± 0.00%)}	_~	_2,872	_2,872	_{p=1.000 n=6}
_{Req 5 - completionInfo}	_{383ms (± 0.98%)}	_{375ms (± 1.01%)}	_{-8ms (- 1.96%)}	_372ms	_380ms	_{p=0.013 n=6}
_{Compiler-UnionsTSServer - node (v16.17.1, x64)}
_{Req 1 - updateOpen}	_{2,629ms (± 0.40%)}	_{2,615ms (± 0.80%)}	_~	_2,591ms	_2,645ms	_{p=0.229 n=6}
_{Req 2 - geterr}	_{6,081ms (± 0.38%)}	_{6,084ms (± 0.67%)}	_~	_6,004ms	_6,116ms	_{p=0.521 n=6}
_{Req 3 - references}	_{355ms (± 0.37%)}	_{356ms (± 0.66%)}	_~	_353ms	_360ms	_{p=0.217 n=6}
_{Req 4 - navto}	_{284ms (± 0.41%)}	_{284ms (± 0.84%)}	_~	_281ms	_288ms	_{p=0.934 n=6}
_{Req 5 - completionInfo count}	_{1,356 (± 0.00%)}	_{1,356 (± 0.00%)}	_~	_1,356	_1,356	_{p=1.000 n=6}
_{Req 5 - completionInfo}	_{83ms (± 7.49%)}	_{81ms (± 1.22%)}	_~	_80ms	_82ms	_{p=0.673 n=6}
_{CompilerTSServer - node (v16.17.1, x64)}
_{Req 1 - updateOpen}	_{2,803ms (± 0.52%)}	_{2,798ms (± 0.89%)}	_~	_2,760ms	_2,824ms	_{p=0.936 n=6}
_{Req 2 - geterr}	_{4,689ms (± 0.44%)}	_{4,699ms (± 0.36%)}	_~	_4,673ms	_4,723ms	_{p=0.470 n=6}
_{Req 3 - references}	_{364ms (± 0.23%)}	_{363ms (± 0.58%)}	_~	_361ms	_366ms	_{p=0.505 n=6}
_{Req 4 - navto}	_{281ms (± 0.95%)}	_{281ms (± 0.84%)}	_~	_278ms	_284ms	_{p=1.000 n=6}
_{Req 5 - completionInfo count}	_{1,518 (± 0.00%)}	_{1,518 (± 0.00%)}	_~	_1,518	_1,518	_{p=1.000 n=6}
_{Req 5 - completionInfo}	_{77ms (± 1.09%)}	_{76ms (± 1.75%)}	_~	_75ms	_78ms	_{p=0.342 n=6}
_{xstateTSServer - node (v16.17.1, x64)}
_{Req 1 - updateOpen}	_{3,230ms (± 0.30%)}	_{3,221ms (± 0.24%)}	_~	_3,211ms	_3,229ms	_{p=0.261 n=6}
_{Req 2 - geterr}	_{1,733ms (± 1.20%)}	_{1,757ms (± 1.34%)}	_~	_1,728ms	_1,788ms	_{p=0.093 n=6}
_{Req 3 - references}	_{130ms (± 7.76%)}	_{126ms (± 6.65%)}	_~	_121ms	_143ms	_{p=0.216 n=6}
_{Req 4 - navto}	_{352ms (± 0.49%)}	_{351ms (± 0.28%)}	_~	_350ms	_352ms	_{p=0.510 n=6}
_{Req 5 - completionInfo count}	_{2,872 (± 0.00%)}	_{2,872 (± 0.00%)}	_~	_2,872	_2,872	_{p=1.000 n=6}
_{Req 5 - completionInfo}	_{429ms (± 0.71%)}	_{423ms (± 1.47%)}	_~	_414ms	_431ms	_{p=0.092 n=6}
_{Compiler-UnionsTSServer - node (v14.21.3, x64)}
_{Req 1 - updateOpen}	_{2,771ms (± 0.36%)}	_{2,767ms (± 0.33%)}	_~	_2,750ms	_2,774ms	_{p=0.688 n=6}
_{Req 2 - geterr}	_{6,220ms (± 0.51%)}	_{6,188ms (± 0.65%)}	_~	_6,122ms	_6,247ms	_{p=0.298 n=6}
_{Req 3 - references}	_{361ms (± 0.51%)}	_{364ms (± 0.51%)}	_~	_362ms	_367ms	_{p=0.061 n=6}
_{Req 4 - navto}	_{291ms (± 0.92%)}	_{291ms (± 0.78%)}	_~	_289ms	_295ms	_{p=1.000 n=6}
_{Req 5 - completionInfo count}	_{1,356 (± 0.00%)}	_{1,356 (± 0.00%)}	_~	_1,356	_1,356	_{p=1.000 n=6}
_{Req 5 - completionInfo}	_{102ms (± 8.45%)}	_{102ms (± 9.41%)}	_~	_91ms	_111ms	_{p=1.000 n=6}
_{CompilerTSServer - node (v14.21.3, x64)}
_{Req 1 - updateOpen}	_{2,936ms (± 0.28%)}	_{2,931ms (± 0.49%)}	_~	_2,906ms	_2,943ms	_{p=0.520 n=6}
_{Req 2 - geterr}	_{4,583ms (± 0.44%)}	_{4,573ms (± 0.48%)}	_~	_4,535ms	_4,594ms	_{p=0.575 n=6}
_{Req 3 - references}	_{370ms (± 0.76%)}	_{369ms (± 0.68%)}	_~	_366ms	_372ms	_{p=0.629 n=6}
_{Req 4 - navto}	_{299ms (± 0.46%)}	_{298ms (± 0.25%)}	_~	_297ms	_299ms	_{p=0.111 n=6}
_{Req 5 - completionInfo count}	_{1,518 (± 0.00%)}	_{1,518 (± 0.00%)}	_~	_1,518	_1,518	_{p=1.000 n=6}
_{Req 5 - completionInfo}	_{86ms (± 8.01%)}	_{87ms (± 7.63%)}	_~	_82ms	_100ms	_{p=0.366 n=6}
_{xstateTSServer - node (v14.21.3, x64)}
_{Req 1 - updateOpen}	_{3,498ms (± 1.79%)}	_{3,514ms (± 0.78%)}	_~	_3,480ms	_3,538ms	_{p=1.000 n=6}
_{Req 2 - geterr}	_{1,863ms (± 0.80%)}	_{1,899ms (± 0.82%)}	_{+36ms (+ 1.91%)}	_1,876ms	_1,919ms	_{p=0.008 n=6}
_{Req 3 - references}	_{139ms (± 8.58%)}	_{135ms (± 6.70%)}	_~	_130ms	_153ms	_{p=0.623 n=6}
_{Req 4 - navto}	_{391ms (± 0.77%)}	_{386ms (± 0.16%)}	_{-5ms (- 1.15%)}	_385ms	_387ms	_{p=0.005 n=6}
_{Req 5 - completionInfo count}	_{2,872 (± 0.00%)}	_{2,872 (± 0.00%)}	_~	_2,872	_2,872	_{p=1.000 n=6}
_{Req 5 - completionInfo}	_{420ms (± 1.35%)}	_{420ms (± 0.77%)}	_~	_415ms	_424ms	_{p=1.000 n=6}

System

_{Machine Name}	_ts-ci-ubuntu
_Platform	_{linux 5.4.0-148-generic}
_Architecture	_x64
_{Available Memory}	_{16 GB}
_{Available Memory}	_{15 GB}
_CPUs	_{4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz}

Hosts

node (v18.10.0, x64)
node (v16.17.1, x64)
node (v14.21.3, x64)

Scenarios

Compiler-UnionsTSServer - node (v18.10.0, x64)
Compiler-UnionsTSServer - node (v16.17.1, x64)
Compiler-UnionsTSServer - node (v14.21.3, x64)
CompilerTSServer - node (v18.10.0, x64)
CompilerTSServer - node (v16.17.1, x64)
CompilerTSServer - node (v14.21.3, x64)
xstateTSServer - node (v18.10.0, x64)
xstateTSServer - node (v16.17.1, x64)
xstateTSServer - node (v14.21.3, x64)

_Benchmark	_Name	_Iterations
_Current	₅₅₂₂₂	₆
_Baseline	_main	₆

Startup

Comparison Report - main..55222

_Metric	_main	₅₅₂₂₂	_Delta	_Best	_Worst	_p-value
_{tsc-startup - node (v16.17.1, x64)}
_{Execution time}	_{143.04ms (± 0.25%)}	_{142.52ms (± 0.18%)}	_{-0.52ms (- 0.37%)}	_141.86ms	_145.01ms	_{p=0.000 n=600}
_{tsserver-startup - node (v16.17.1, x64)}
_{Execution time}	_{222.23ms (± 0.17%)}	_{221.80ms (± 0.18%)}	_{-0.43ms (- 0.19%)}	_220.94ms	_229.73ms	_{p=0.000 n=600}
_{tsserverlibrary-startup - node (v16.17.1, x64)}
_{Execution time}	_{223.71ms (± 0.17%)}	_{223.26ms (± 0.17%)}	_{-0.45ms (- 0.20%)}	_222.31ms	_228.31ms	_{p=0.000 n=600}
_{typescript-startup - node (v16.17.1, x64)}
_{Execution time}	_{206.38ms (± 0.24%)}	_{205.27ms (± 0.16%)}	_{-1.10ms (- 0.53%)}	_204.37ms	_209.04ms	_{p=0.000 n=600}

System

_{Machine Name}	_ts-ci-ubuntu
_Platform	_{linux 5.4.0-148-generic}
_Architecture	_x64
_{Available Memory}	_{16 GB}
_{Available Memory}	_{15 GB}
_CPUs	_{4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz}

Hosts

node (v16.17.1, x64)

Scenarios

tsc-startup - node (v16.17.1, x64)
tsserver-startup - node (v16.17.1, x64)
tsserverlibrary-startup - node (v16.17.1, x64)
typescript-startup - node (v16.17.1, x64)

_Benchmark	_Name	_Iterations
_Current	₅₅₂₂₂	₆
_Baseline	_main	₆

Developer Information:

Download Benchmark

Aug 01 '23 01:08 typescript-bot

@jakebailey Here are the results of running the top-repos suite comparing main and refs/pull/55222/merge:

Everything looks good!

Aug 01 '23 01:08 typescript-bot

Hey @jakebailey, it looks like the DT test run failed. Please check the log for more details. You can check the log here.

Aug 01 '23 01:08 typescript-bot

@jakebailey Here are the results of running the top-repos suite comparing main and refs/pull/55222/merge:

Everything looks good!

Aug 01 '23 02:08 typescript-bot

7% check degradation in xstate, ouch

@typescript-bot perf test this faster

Aug 01 '23 03:08 jakebailey

Heya @jakebailey, I've started to run the abridged perf test suite on this PR at 59020188c759576c1212fcfc9a213bcdd5436ca9. You can monitor the build here.

Update: The results are in!

Aug 01 '23 03:08 typescript-bot

@jakebailey The results of the perf run you requested are in!

Here they are:

Comparison Report - main..55222

_Metric	_main	₅₅₂₂₂	_Delta	_Best	_Worst	_p-value
_{Angular - node (v16.17.1, x64)}
_{Memory used}	_{368,218k (± 0.00%)}	_{368,212k (± 0.00%)}	_~	_368,196k	_368,238k	_{p=0.298 n=6}
_{Parse Time}	_{3.57s (± 0.39%)}	_{3.58s (± 0.63%)}	_~	_3.54s	_3.60s	_{p=0.370 n=6}
_{Bind Time}	_{1.19s (± 0.63%)}	_{1.19s (± 0.34%)}	_~	_1.19s	_1.20s	_{p=0.389 n=6}
_{Check Time}	_{9.76s (± 0.72%)}	_{9.73s (± 0.32%)}	_~	_9.68s	_9.77s	_{p=0.294 n=6}
_{Emit Time}	_{8.04s (± 0.80%)}	_{7.99s (± 0.62%)}	_~	_7.92s	_8.04s	_{p=0.107 n=6}
_{Total Time}	_{22.55s (± 0.58%)}	_{22.48s (± 0.36%)}	_~	_22.40s	_22.58s	_{p=0.520 n=6}
_{Compiler-Unions - node (v16.17.1, x64)}
_{Memory used}	_{193,823k (± 0.02%)}	_{193,817k (± 0.01%)}	_~	_193,777k	_193,851k	_{p=0.936 n=6}
_{Parse Time}	_{1.59s (± 0.96%)}	_{1.59s (± 0.86%)}	_~	_1.56s	_1.60s	_{p=0.867 n=6}
_{Bind Time}	_{0.82s (± 0.92%)}	_{0.82s (± 0.99%)}	_~	_0.82s	_0.84s	_{p=1.000 n=6}
_{Check Time}	_{10.16s (± 0.48%)}	_{10.14s (± 0.76%)}	_~	_10.04s	_10.23s	_{p=0.628 n=6}
_{Emit Time}	_{2.98s (± 0.87%)}	_{2.97s (± 0.45%)}	_~	_2.95s	_2.98s	_{p=0.565 n=6}
_{Total Time}	_{15.56s (± 0.35%)}	_{15.52s (± 0.54%)}	_~	_15.39s	_15.60s	_{p=0.423 n=6}
_{Monaco - node (v16.17.1, x64)}
_{Memory used}	_{347,078k (± 0.01%)}	_{347,080k (± 0.01%)}	_~	_347,050k	_347,102k	_{p=0.688 n=6}
_{Parse Time}	_{2.77s (± 0.44%)}	_{2.77s (± 0.63%)}	_~	_2.75s	_2.79s	_{p=0.369 n=6}
_{Bind Time}	_{1.08s (± 0.48%)}	_{1.08s (± 0.59%)}	_~	_1.07s	_1.09s	_{p=0.386 n=6}
_{Check Time}	_{8.04s (± 0.61%)}	_{8.01s (± 0.37%)}	_~	_7.96s	_8.05s	_{p=0.291 n=6}
_{Emit Time}	_{4.48s (± 0.56%)}	_{4.45s (± 0.59%)}	_~	_4.42s	_4.49s	_{p=0.063 n=6}
_{Total Time}	_{16.36s (± 0.36%)}	_{16.31s (± 0.38%)}	_~	_16.25s	_16.42s	_{p=0.128 n=6}
_{TFS - node (v16.17.1, x64)}
_{Memory used}	_{301,118k (± 0.01%)}	_{301,102k (± 0.01%)}	_~	_301,080k	_301,137k	_{p=0.199 n=6}
_{Parse Time}	_{2.20s (± 0.38%)}	_{2.21s (± 0.62%)}	_{+0.02s (+ 0.84%)}	_2.20s	_2.23s	_{p=0.021 n=6}
_{Bind Time}	_{1.22s (± 1.70%)}	_{1.22s (± 0.84%)}	_~	_1.21s	_1.24s	_{p=0.459 n=6}
_{Check Time}	_{7.34s (± 0.31%)}	_{7.33s (± 0.22%)}	_~	_7.31s	_7.35s	_{p=0.252 n=6}
_{Emit Time}	_{4.30s (± 1.17%)}	_{4.29s (± 0.41%)}	_~	_4.27s	_4.32s	_{p=0.745 n=6}
_{Total Time}	_{15.06s (± 0.25%)}	_{15.06s (± 0.30%)}	_~	_15.00s	_15.11s	_{p=0.871 n=6}
_{material-ui - node (v16.17.1, x64)}
_{Memory used}	_{481,882k (± 0.01%)}	_{481,875k (± 0.01%)}	_~	_481,830k	_481,901k	_{p=0.748 n=6}
_{Parse Time}	_{3.26s (± 0.72%)}	_{3.27s (± 0.45%)}	_~	_3.25s	_3.29s	_{p=0.287 n=6}
_{Bind Time}	_{0.96s (± 0.85%)}	_{0.95s (± 0.54%)}	_~	_0.95s	_0.96s	_{p=0.523 n=6}
_{Check Time}	_{18.34s (± 0.39%)}	_{18.35s (± 0.65%)}	_~	_18.20s	_18.52s	_{p=0.936 n=6}
_{Emit Time}	_{0.00s (± 0.00%)}	_{0.00s (± 0.00%)}	_~	_0.00s	_0.00s	_{p=1.000 n=6}
_{Total Time}	_{22.55s (± 0.36%)}	_{22.58s (± 0.51%)}	_~	_22.45s	_22.75s	_{p=0.809 n=6}
_{xstate - node (v16.17.1, x64)}
_{Memory used}	_{561,343k (± 0.01%)}	_{563,176k (± 0.04%)}	_{+1,833k (+ 0.33%)}	_562,997k	_563,455k	_{p=0.005 n=6}
_{Parse Time}	_{4.02s (± 0.30%)}	_{4.02s (± 0.30%)}	_~	_4.01s	_4.04s	_{p=1.000 n=6}
_{Bind Time}	_{1.66s (± 5.88%)}	_{1.72s (± 5.38%)}	_~	_1.54s	_1.80s	_{p=0.370 n=6}
_{Check Time}	_{3.09s (± 1.60%)}	_{3.31s (± 2.99%)}	_{+0.22s (+ 7.06%)}	_3.25s	_3.51s	_{p=0.005 n=6}
_{Emit Time}	_{0.09s (± 5.53%)}	_{0.09s (± 0.00%)}	_~	_0.09s	_0.09s	_{p=0.174 n=6}
_{Total Time}	_{8.87s (± 0.58%)}	_{9.15s (± 0.21%)}	_{+0.28s (+ 3.16%)}	_9.12s	_9.17s	_{p=0.005 n=6}

System

_{Machine Name}	_ts-ci-ubuntu
_Platform	_{linux 5.4.0-148-generic}
_Architecture	_x64
_{Available Memory}	_{16 GB}
_{Available Memory}	_{15 GB}
_CPUs	_{4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz}

Hosts

node (v16.17.1, x64)

Scenarios

Angular - node (v16.17.1, x64)
Compiler-Unions - node (v16.17.1, x64)
Monaco - node (v16.17.1, x64)
TFS - node (v16.17.1, x64)
material-ui - node (v16.17.1, x64)
xstate - node (v16.17.1, x64)

_Benchmark	_Name	_Iterations
_Current	₅₅₂₂₂	₆
_Baseline	_main	₆

Developer Information:

Download Benchmark

Aug 01 '23 03:08 typescript-bot

This PR failed RWC but the diffs aren't correctly being pushed. Need to figure that out.

Aug 01 '23 22:08 jakebailey

@typescript-bot perf test this faster

Aug 15 '23 17:08 jakebailey

Heya @jakebailey, I've started to run the tsc-only perf test suite on this PR at 59020188c759576c1212fcfc9a213bcdd5436ca9. You can monitor the build here.

Update: The results are in!

Aug 15 '23 17:08 typescript-bot

@jakebailey The results of the perf run you requested are in!

Here they are:

Compiler

Comparison Report - baseline..pr

_Metric	_baseline	_pr	_Delta	_Best	_Worst	_p-value
_{Angular - node (v16.17.1, x64)}
_{Memory used}	_{300,307k (± 0.01%)}	_{300,336k (± 0.00%)}	_{+30k (+ 0.01%)}	_300,312k	_300,349k	_{p=0.020 n=6}
_{Parse Time}	_{3.02s (± 0.13%)}	_{3.03s (± 0.13%)}	_{+0.01s (+ 0.22%)}	_3.02s	_3.03s	_{p=0.034 n=6}
_{Bind Time}	_{0.93s (± 0.00%)}	_{0.93s (± 0.00%)}	_~	_0.93s	_0.93s	_{p=1.000 n=6}
_{Check Time}	_{9.46s (± 0.26%)}	_{9.45s (± 0.20%)}	_~	_9.43s	_9.47s	_{p=0.188 n=6}
_{Emit Time}	_{7.62s (± 0.29%)}	_{7.63s (± 0.41%)}	_~	_7.59s	_7.68s	_{p=1.000 n=6}
_{Total Time}	_{21.04s (± 0.12%)}	_{21.03s (± 0.15%)}	_~	_21.00s	_21.07s	_{p=0.935 n=6}
_{Compiler-Unions - node (v16.17.1, x64)}
_{Memory used}	_{193,822k (± 0.01%)}	_{193,830k (± 0.01%)}	_~	_193,797k	_193,859k	_{p=0.810 n=6}
_{Parse Time}	_{1.57s (± 1.05%)}	_{1.56s (± 1.05%)}	_~	_1.55s	_1.59s	_{p=0.441 n=6}
_{Bind Time}	_{0.79s (± 0.00%)}	_{0.79s (± 0.00%)}	_~	_0.79s	_0.79s	_{p=1.000 n=6}
_{Check Time}	_{9.90s (± 0.21%)}	_{9.86s (± 0.17%)}	_{-0.04s (- 0.37%)}	_9.84s	_9.88s	_{p=0.012 n=6}
_{Emit Time}	_{2.75s (± 0.19%)}	_{2.74s (± 0.38%)}	_~	_2.73s	_2.76s	_{p=0.437 n=6}
_{Total Time}	_{15.01s (± 0.21%)}	_{14.95s (± 0.18%)}	_{-0.06s (- 0.38%)}	_14.91s	_14.99s	_{p=0.013 n=6}
_{Monaco - node (v16.17.1, x64)}
_{Memory used}	_{347,109k (± 0.01%)}	_{347,102k (± 0.01%)}	_~	_347,081k	_347,129k	_{p=0.575 n=6}
_{Parse Time}	_{2.68s (± 0.30%)}	_{2.69s (± 0.39%)}	_~	_2.67s	_2.70s	_{p=0.865 n=6}
_{Bind Time}	_{0.99s (± 0.00%)}	_{0.99s (± 0.00%)}	_~	_0.99s	_0.99s	_{p=1.000 n=6}
_{Check Time}	_{7.90s (± 0.38%)}	_{7.91s (± 0.13%)}	_~	_7.89s	_7.92s	_{p=0.806 n=6}
_{Emit Time}	_{4.25s (± 0.28%)}	_{4.26s (± 0.38%)}	_~	_4.23s	_4.28s	_{p=0.549 n=6}
_{Total Time}	_{15.83s (± 0.19%)}	_{15.83s (± 0.17%)}	_~	_15.80s	_15.88s	_{p=0.570 n=6}
_{TFS - node (v16.17.1, x64)}
_{Memory used}	_{301,119k (± 0.01%)}	_{301,119k (± 0.00%)}	_~	_301,101k	_301,135k	_{p=1.000 n=6}
_{Parse Time}	_{2.18s (± 0.61%)}	_{2.18s (± 0.56%)}	_~	_2.16s	_2.19s	_{p=0.863 n=6}
_{Bind Time}	_{1.11s (± 1.51%)}	_{1.11s (± 0.57%)}	_~	_1.10s	_1.12s	_{p=0.720 n=6}
_{Check Time}	_{7.22s (± 0.40%)}	_{7.22s (± 0.32%)}	_~	_7.19s	_7.26s	_{p=1.000 n=6}
_{Emit Time}	_{3.97s (± 0.23%)}	_{3.97s (± 0.36%)}	_~	_3.95s	_3.99s	_{p=1.000 n=6}
_{Total Time}	_{14.48s (± 0.33%)}	_{14.48s (± 0.13%)}	_~	_14.46s	_14.51s	_{p=0.687 n=6}
_{material-ui - node (v16.17.1, x64)}
_{Memory used}	_{479,389k (± 0.01%)}	_{479,417k (± 0.01%)}	_~	_479,335k	_479,503k	_{p=0.575 n=6}
_{Parse Time}	_{3.15s (± 0.24%)}	_{3.15s (± 0.13%)}	_~	_3.15s	_3.16s	_{p=0.389 n=6}
_{Bind Time}	_{0.91s (± 0.45%)}	_{0.91s (± 0.00%)}	_~	_0.91s	_0.91s	_{p=0.405 n=6}
_{Check Time}	_{17.90s (± 0.25%)}	_{17.92s (± 0.33%)}	_~	_17.85s	_18.00s	_{p=0.872 n=6}
_{Emit Time}	_{0.00s (± 0.00%)}	_{0.00s (± 0.00%)}	_~	_0.00s	_0.00s	_{p=1.000 n=6}
_{Total Time}	_{21.96s (± 0.21%)}	_{21.98s (± 0.25%)}	_~	_21.91s	_22.05s	_{p=0.748 n=6}
_{xstate - node (v16.17.1, x64)}
_{Memory used}	_{542,764k (± 0.01%)}	_{544,461k (± 0.01%)}	_{+1,696k (+ 0.31%)}	_544,400k	_544,557k	_{p=0.005 n=6}
_{Parse Time}	_{3.70s (± 0.17%)}	_{3.70s (± 0.17%)}	_~	_3.69s	_3.71s	_{p=1.000 n=6}
_{Bind Time}	_{1.38s (± 4.67%)}	_{1.40s (± 4.45%)}	_~	_1.34s	_1.46s	_{p=1.000 n=6}
_{Check Time}	_{3.30s (± 2.68%)}	_{3.41s (± 2.21%)}	_~	_3.33s	_3.49s	_{p=0.128 n=6}
_{Emit Time}	_{0.08s (± 0.00%)}	_{0.08s (± 6.19%)}	_~	_0.08s	_0.09s	_{p=0.174 n=6}
_{Total Time}	_{8.46s (± 0.38%)}	_{8.58s (± 0.23%)}	_{+0.13s (+ 1.48%)}	_8.55s	_8.60s	_{p=0.005 n=6}

System info unknown

Hosts

node (v16.17.1, x64)

Scenarios

Angular - node (v16.17.1, x64)
Compiler-Unions - node (v16.17.1, x64)
Monaco - node (v16.17.1, x64)
TFS - node (v16.17.1, x64)
material-ui - node (v16.17.1, x64)
xstate - node (v16.17.1, x64)

_Benchmark	_Name	_Iterations
_Current	_pr	₆
_Baseline	_baseline	₆

Developer Information:

Download Benchmarks

Aug 15 '23 17:08 typescript-bot

Hm, maybe it isn't actually bad?

Aug 15 '23 17:08 jakebailey

@weswigham curious if you have any thoughts on this one.

Aug 15 '23 17:08 jakebailey

@typescript-bot perf test this

Sep 06 '23 23:09 jakebailey

Heya @jakebailey, I've started to run the regular perf test suite on this PR at 540f073561ba27f121ea95b00b6a83a912bfbfdf. You can monitor the build here.

Update: The results are in!

Sep 06 '23 23:09 typescript-bot

@jakebailey The results of the perf run you requested are in!

Here they are:

Compiler

Comparison Report - baseline..pr

_Metric	_baseline	_pr	_Delta	_Best	_Worst	_p-value
_{Angular - node (v16.17.1, x64)}
_{Memory used}	_{300,268k (± 0.01%)}	_{300,270k (± 0.01%)}	_~	_300,252k	_300,288k	_{p=0.872 n=6}
_{Parse Time}	_{3.01s (± 0.17%)}	_{3.00s (± 0.17%)}	_~	_3.00s	_3.01s	_{p=0.311 n=6}
_{Bind Time}	_{0.93s (± 0.00%)}	_{0.93s (± 0.44%)}	_~	_0.92s	_0.93s	_{p=0.405 n=6}
_{Check Time}	_{9.31s (± 0.21%)}	_{9.31s (± 0.23%)}	_~	_9.28s	_9.34s	_{p=0.516 n=6}
_{Emit Time}	_{7.63s (± 0.21%)}	_{7.64s (± 0.43%)}	_~	_7.59s	_7.67s	_{p=0.744 n=6}
_{Total Time}	_{20.88s (± 0.13%)}	_{20.89s (± 0.18%)}	_~	_20.85s	_20.95s	_{p=1.000 n=6}
_{Compiler-Unions - node (v16.17.1, x64)}
_{Memory used}	_{193,910k (± 0.02%)}	_{194,473k (± 0.66%)}	_~	_193,886k	_197,107k	_{p=0.230 n=6}
_{Parse Time}	_{1.58s (± 0.53%)}	_{1.58s (± 0.00%)}	_~	_1.58s	_1.58s	_{p=0.176 n=6}
_{Bind Time}	_{0.79s (± 0.65%)}	_{0.79s (± 0.69%)}	_~	_0.79s	_0.80s	_{p=0.640 n=6}
_{Check Time}	_{9.96s (± 0.75%)}	_{9.92s (± 0.24%)}	_~	_9.89s	_9.95s	_{p=0.627 n=6}
_{Emit Time}	_{2.74s (± 0.27%)}	_{2.74s (± 0.30%)}	_~	_2.74s	_2.76s	_{p=0.652 n=6}
_{Total Time}	_{15.07s (± 0.50%)}	_{15.05s (± 0.20%)}	_~	_15.01s	_15.09s	_{p=0.745 n=6}
_{Monaco - node (v16.17.1, x64)}
_{Memory used}	_{347,172k (± 0.01%)}	_{347,159k (± 0.01%)}	_~	_347,117k	_347,201k	_{p=0.630 n=6}
_{Parse Time}	_{2.68s (± 0.19%)}	_{2.68s (± 0.28%)}	_~	_2.67s	_2.69s	_{p=0.784 n=6}
_{Bind Time}	_{0.99s (± 0.00%)}	_{0.99s (± 0.41%)}	_~	_0.98s	_0.99s	_{p=0.405 n=6}
_{Check Time}	_{7.92s (± 0.33%)}	_{7.92s (± 0.35%)}	_~	_7.89s	_7.96s	_{p=0.935 n=6}
_{Emit Time}	_{4.26s (± 0.30%)}	_{4.26s (± 0.27%)}	_~	_4.25s	_4.28s	_{p=1.000 n=6}
_{Total Time}	_{15.85s (± 0.19%)}	_{15.86s (± 0.19%)}	_~	_15.82s	_15.89s	_{p=1.000 n=6}
_{TFS - node (v16.17.1, x64)}
_{Memory used}	_{301,175k (± 0.00%)}	_{301,172k (± 0.00%)}	_~	_301,146k	_301,187k	_{p=0.936 n=6}
_{Parse Time}	_{2.17s (± 0.77%)}	_{2.18s (± 0.45%)}	_~	_2.17s	_2.20s	_{p=0.127 n=6}
_{Bind Time}	_{1.11s (± 0.46%)}	_{1.11s (± 0.37%)}	_~	_1.11s	_1.12s	_{p=0.595 n=6}
_{Check Time}	_{7.23s (± 0.26%)}	_{7.22s (± 0.38%)}	_~	_7.19s	_7.26s	_{p=0.936 n=6}
_{Emit Time}	_{3.98s (± 0.41%)}	_{3.97s (± 0.34%)}	_~	_3.95s	_3.99s	_{p=0.414 n=6}
_{Total Time}	_{14.48s (± 0.21%)}	_{14.49s (± 0.19%)}	_~	_14.44s	_14.52s	_{p=0.628 n=6}
_{material-ui - node (v16.17.1, x64)}
_{Memory used}	_{479,471k (± 0.00%)}	_{479,473k (± 0.00%)}	_~	_479,466k	_479,485k	_{p=0.747 n=6}
_{Parse Time}	_{3.15s (± 0.00%)}	_{3.15s (± 0.26%)}	_~	_3.14s	_3.16s	_{p=0.290 n=6}
_{Bind Time}	_{0.91s (± 0.00%)}	_{0.91s (± 0.00%)}	_~	_0.91s	_0.91s	_{p=1.000 n=6}
_{Check Time}	_{17.80s (± 0.40%)}	_{17.83s (± 0.40%)}	_~	_17.75s	_17.90s	_{p=0.627 n=6}
_{Emit Time}	_{0.00s (± 0.00%)}	_{0.00s (± 0.00%)}	_~	_0.00s	_0.00s	_{p=1.000 n=6}
_{Total Time}	_{21.86s (± 0.32%)}	_{21.89s (± 0.33%)}	_~	_21.81s	_21.97s	_{p=0.517 n=6}
_{xstate - node (v16.17.1, x64)}
_{Memory used}	_{542,837k (± 0.01%)}	_{544,554k (± 0.02%)}	_{+1,717k (+ 0.32%)}	_544,454k	_544,744k	_{p=0.005 n=6}
_{Parse Time}	_{3.68s (± 0.37%)}	_{3.69s (± 0.37%)}	_~	_3.67s	_3.70s	_{p=0.935 n=6}
_{Bind Time}	_{1.42s (± 6.89%)}	_{1.44s (± 3.31%)}	_~	_1.34s	_1.46s	_{p=0.865 n=6}
_{Check Time}	_{3.24s (± 3.02%)}	_{3.39s (± 2.07%)}	_{+0.15s (+ 4.58%)}	_3.34s	_3.53s	_{p=0.010 n=6}
_{Emit Time}	_{0.08s (± 6.44%)}	_{0.08s (± 4.99%)}	_~	_0.08s	_0.09s	_{p=0.441 n=6}
_{Total Time}	_{8.43s (± 0.56%)}	_{8.60s (± 0.39%)}	_{+0.17s (+ 2.02%)}	_8.57s	_8.66s	_{p=0.005 n=6}

System info unknown

Hosts

node (v16.17.1, x64)

Scenarios

Angular - node (v16.17.1, x64)
Compiler-Unions - node (v16.17.1, x64)
Monaco - node (v16.17.1, x64)
TFS - node (v16.17.1, x64)
material-ui - node (v16.17.1, x64)
xstate - node (v16.17.1, x64)

_Benchmark	_Name	_Iterations
_Current	_pr	₆
_Baseline	_baseline	₆

tsserver

Comparison Report - baseline..pr

_Metric	_baseline	_pr	_Delta	_Best	_Worst	_p-value
_{Compiler-UnionsTSServer - node (v16.17.1, x64)}
_{Req 1 - updateOpen}	_{2,490ms (± 0.11%)}	_{2,492ms (± 0.18%)}	_~	_2,486ms	_2,500ms	_{p=0.512 n=6}
_{Req 2 - geterr}	_{5,942ms (± 0.44%)}	_{5,926ms (± 0.31%)}	_~	_5,906ms	_5,951ms	_{p=0.298 n=6}
_{Req 3 - references}	_{343ms (± 0.34%)}	_{343ms (± 0.36%)}	_~	_341ms	_344ms	_{p=0.607 n=6}
_{Req 4 - navto}	_{278ms (± 0.54%)}	_{278ms (± 0.29%)}	_~	_277ms	_279ms	_{p=0.498 n=6}
_{Req 5 - completionInfo count}	_{1,356 (± 0.00%)}	_{1,356 (± 0.00%)}	_~	_1,356	_1,356	_{p=1.000 n=6}
_{Req 5 - completionInfo}	_{81ms (± 8.18%)}	_{85ms (± 9.74%)}	_~	_75ms	_93ms	_{p=0.620 n=6}
_{CompilerTSServer - node (v16.17.1, x64)}
_{Req 1 - updateOpen}	_{2,633ms (± 0.54%)}	_{2,634ms (± 0.74%)}	_~	_2,610ms	_2,662ms	_{p=0.936 n=6}
_{Req 2 - geterr}	_{4,772ms (± 0.37%)}	_{4,778ms (± 0.31%)}	_~	_4,757ms	_4,799ms	_{p=0.629 n=6}
_{Req 3 - references}	_{351ms (± 0.24%)}	_{350ms (± 0.21%)}	_~	_349ms	_351ms	_{p=0.652 n=6}
_{Req 4 - navto}	_{270ms (± 0.49%)}	_{270ms (± 0.28%)}	_~	_269ms	_271ms	_{p=0.730 n=6}
_{Req 5 - completionInfo count}	_{1,518 (± 0.00%)}	_{1,518 (± 0.00%)}	_~	_1,518	_1,518	_{p=1.000 n=6}
_{Req 5 - completionInfo}	_{79ms (± 1.03%)}	_{79ms (± 0.00%)}	_~	_79ms	_79ms	_{p=0.405 n=6}
_{xstateTSServer - node (v16.17.1, x64)}
_{Req 1 - updateOpen}	_{2,704ms (± 0.18%)}	_{2,706ms (± 0.21%)}	_~	_2,698ms	_2,712ms	_{p=0.809 n=6}
_{Req 2 - geterr}	_{1,976ms (± 0.54%)}	_{1,946ms (± 2.24%)}	_{-30ms (- 1.49%)}	_1,857ms	_1,966ms	_{p=0.024 n=6}
_{Req 3 - references}	_{139ms (± 2.40%)}	_{139ms (± 2.20%)}	_~	_134ms	_143ms	_{p=1.000 n=6}
_{Req 4 - navto}	_{352ms (± 0.28%)}	_{353ms (± 0.35%)}	_~	_351ms	_354ms	_{p=0.359 n=6}
_{Req 5 - completionInfo count}	_{2,071 (± 0.00%)}	_{2,071 (± 0.00%)}	_~	_2,071	_2,071	_{p=1.000 n=6}
_{Req 5 - completionInfo}	_{321ms (± 2.05%)}	_{323ms (± 1.95%)}	_~	_313ms	_328ms	_{p=0.325 n=6}

System info unknown

Hosts

node (v16.17.1, x64)

Scenarios

CompilerTSServer - node (v16.17.1, x64)
Compiler-UnionsTSServer - node (v16.17.1, x64)
xstateTSServer - node (v16.17.1, x64)

_Benchmark	_Name	_Iterations
_Current	_pr	₆
_Baseline	_baseline	₆

Startup

Comparison Report - baseline..pr

_Metric	_baseline	_pr	_Delta	_Best	_Worst	_p-value
_{tsc-startup - node (v16.17.1, x64)}
_{Execution time}	_{156.31ms (± 0.15%)}	_{156.36ms (± 0.15%)}	_{+0.05ms (+ 0.03%)}	_154.75ms	_158.84ms	_{p=0.003 n=600}
_{tsserver-startup - node (v16.17.1, x64)}
_{Execution time}	_{230.51ms (± 0.14%)}	_{231.18ms (± 0.16%)}	_{+0.67ms (+ 0.29%)}	_229.52ms	_236.41ms	_{p=0.000 n=600}
_{tsserverlibrary-startup - node (v16.17.1, x64)}
_{Execution time}	_{235.25ms (± 0.11%)}	_{236.07ms (± 0.11%)}	_{+0.82ms (+ 0.35%)}	_234.39ms	_240.84ms	_{p=0.000 n=600}
_{typescript-startup - node (v16.17.1, x64)}
_{Execution time}	_{236.20ms (± 0.16%)}	_{236.01ms (± 0.13%)}	_{-0.19ms (- 0.08%)}	_234.36ms	_240.88ms	_{p=0.000 n=600}

System info unknown

Hosts

node (v16.17.1, x64)

Scenarios

tsc-startup - node (v16.17.1, x64)
tsserver-startup - node (v16.17.1, x64)
tsserverlibrary-startup - node (v16.17.1, x64)
typescript-startup - node (v16.17.1, x64)

_Benchmark	_Name	_Iterations
_Current	_pr	₆
_Baseline	_baseline	₆

Developer Information:

Download Benchmarks

Sep 07 '23 00:09 typescript-bot

@typescript-bot perf test this faster

Sep 07 '23 00:09 jakebailey

Heya @jakebailey, I've started to run the tsc-only perf test suite on this PR at 540f073561ba27f121ea95b00b6a83a912bfbfdf. You can monitor the build here.

Update: The results are in!

Sep 07 '23 00:09 typescript-bot

Labels

Experiment

Author: Team

For Milestone Bug

For Backlog Bug

Owner

Other Repo Issues