TypeScript
TypeScript copied to clipboard
microsoft
TSC Regression causing crash (Heap OOM) between version 4.8.4 & 4.9.3 in large project
Bug Report
When updating typescript within our project from version 4.7.3 to a more recent version, our project began experiencing OOM issues.
Specifically, it is completely unable to build via tsc or even tsc --noEmit.
Ours is a somewhat large project, so I couldn't point to a minimal reproduction (sorry!).
I have 16GB of RAM so I doubt its a "classic" memory error, seems more likely something started infinitely using memory during a version upgrade.
<--- Last few GCs --->
[7915:0x5f6b230] 32781 ms: Mark-sweep (reduce) 2041.6 (2082.5) -> 2040.8 (2083.0) MB, 492.6 / 0.0 ms (average mu = 0.088, current mu = 0.038) allocation failure; scavenge might not succeed
[7915:0x5f6b230] 33342 ms: Mark-sweep (reduce) 2041.9 (2083.0) -> 2041.1 (2083.2) MB, 556.2 / 0.0 ms (average mu = 0.047, current mu = 0.009) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xb6d470 node::Abort() [node]
2: 0xa7e0a8 [node]
3: 0xd46ee0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xd47287 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xf24645 [node]
6: 0xf25548 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
7: 0xf35a53 [node]
8: 0xf368c8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
9: 0xf1122e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xf125f7 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
11: 0xef37ca v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
12: 0x12b6d6f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
13: 0x16e89b9 [node]
[1] 7915 abort tsc --noEmit
🔎 Search Terms
tsc 4.9 4.9.5 Out of Memory Heap Out of Memory Crash Killed Silent Error (Am i doing this right?)
🕗 Version & Regression Information
Error began occuring when updating from typescript 4.7.3 to any 4.9.3^ version. Issue persists into Version 5.1.0-dev.20230303 (nightly at time of testing). Issue is NOT present in v4.8.4, or any prior version we tried (that satisfied our dependencies, anyways.)
- This is a crash
- This changed between versions 4.8.4 and 4.9.3
💻 Code
Branch state is: https://github.com/Eranziel/foundryvtt-lancer/tree/oom
Was unable to produce a minimally viable reproduction, as the OOM error gives next to no context as to where the issue occurs or why. Sorry. More than willing to provide more investigative information, but I don't really know what there is to provide.
🙁 Actual behavior
It crashes on more recent versions for no clear reason.
🙂 Expected behavior
I'd rather it didn't.
FYI: You can try https://github.com/microsoft/TypeScript/wiki/Performance-Tracing, it helped us locate the problem in https://github.com/microsoft/TypeScript/issues/50515.
FYI: You can try https://github.com/microsoft/TypeScript/wiki/Performance-Tracing, it helped us locate the problem in #50515.
I will attempt this as soon as i have recruited someone with enough ram to run it without crashing :)
4.8.4 Diagnostic information:
Files: 1023
Lines of Library: 28146
Lines of Definitions: 218302
Lines of TypeScript: 25936
Lines of JavaScript: 0
Lines of JSON: 0
Lines of Other: 0
Nodes of Library: 118224
Nodes of Definitions: 564026
Nodes of TypeScript: 115815
Nodes of JavaScript: 0
Nodes of JSON: 0
Nodes of Other: 0
Identifiers: 301992
Symbols: 865373
Types: 208799
Instantiations: 365698
Memory used: 909729K
Assignability cache size: 27600
Identity cache size: 4422
Subtype cache size: 7237
Strict subtype cache size: 9428
Tracing time: 0.17s
I/O Read time: 0.06s
Parse time: 0.96s
ResolveModule time: 0.08s
ResolveTypeReference time: 0.00s
Program time: 1.27s
Bind time: 0.62s
Check time: 2.85s
printTime time: 0.00s
Emit time: 0.00s
Dump types time: 9.03s
Total time: 4.74s
5.1.0-dev.20230305 Diagnostic information:
Files: 1029
Lines of Library: 29397
Lines of Definitions: 218302
Lines of TypeScript: 25916
Lines of JavaScript: 0
Lines of JSON: 0
Lines of Other: 0
Identifiers: 302941
Symbols: 6242486
Types: 702542
Instantiations: 417693
Memory used: 5496499K
Assignability cache size: 31947
Identity cache size: 4714
Subtype cache size: 5307
Strict subtype cache size: 7579
Tracing time: 0.08s
I/O Read time: 0.04s
Parse time: 0.94s
ResolveModule time: 0.09s
ResolveTypeReference time: 0.00s
Program time: 1.19s
Bind time: 0.49s
Check time: 57.68s
printTime time: 0.00s
Emit time: 0.00s
Dump types time: 7.27s
Total time: 59.37s
Notably there is an about 5x increase in memory usage, as well as a >5x increase in the "Types" and "Symbols".
Now that I can actually build on v10 there several new errors, most interesting among them the following:
src/module/item/license-sheet.ts:37:23 - error TS2590: Expression produces a union type that is too complex to represent.
The line is ```TS for (let d of docs as LancerItem[])
// Find the assoc frame
for (let et of [EntryType.FRAME, EntryType.MECH_SYSTEM, EntryType.MECH_WEAPON, EntryType.WEAPON_MOD]) {
let pack = game.packs.get(`world.${et}`);
if (pack) {
let docs = await pack.getDocuments({ "system.license": license.system.key });
for (let d of docs as LancerItem[]) {
let rank = (d as any).system.license_level as number;
while (unlocks.length <= rank) {
unlocks.push([]);
}
unlocks[rank].push(d);
}
}
}
Where docs is of type:
docs: StoredDocument<LancerActor | Scene | LancerItem | JournalEntry | Macro | Playlist | RollTable>[].
Here's where "it's a big project" sort of bites us, in that I can't really minimally represent these types here in this comment. Each is a unique class, all with a shared parent class.
Maybe this is indicative of large union types being the root of our problem?
...
Observing the trace, the file src/module/comp-builder.ts is responsible for an entire 53 SECONDS of the total 57 second Check Time. It produces no errors.
Unsure how to proceed from here.
i had a quite similar issue recently after upgrading typescript. i can't remember the details, but it seems the problem was that under some circumstances tsc started creating a lot of types internally when asked to infer the type of a (typewise) complicated function. in the end, writing const foo:FunctionType = (it) => ... instead of const foo = (it:FunctionArgs):FunctionReturn => ... solved the issue. maybe something similar is happening here?
Similar regression on 5.0.2. For me, 4.8.3 is ok.
thanks for the exceptional bug report @whitespine!
I guess there's another issue about JavaScript heap out of memory errors over here (about 4.3):
- https://github.com/microsoft/TypeScript/issues/44299
This issue specifically mentions a OOM when going from 4.8 to 4.9; if you are experiencing an OOM that isn't explicitly when you go from 4.8 to 4.9, your issue is most certainly different and should be reported as a new issue with its own repro.
I was able to solve the OOM issue for myself with
NODE_OPTIONS=--max-old-space-size=8192 tsc
Since @Raynos solution is really just a stop gap. What is needed to further this ticket. We also have a large project that works fine on 4.8.4 but cant upgrade to 4.9 or 5.0 because tsc gets an OOM error
Here is what I have gathered from running some test.
Test Ran with
NODE_OPTIONS=--max-old-space-size=8192 tsc --noEmit --diagnostics --extendedDiagnostics --incremental false
Here is 4.8
Files: 2979
Lines of Library: 33311
Lines of Definitions: 233669
Lines of TypeScript: 59133
Lines of JavaScript: 0
Lines of JSON: 943
Lines of Other: 0
Nodes of Library: 148246
Nodes of Definitions: 723896
Nodes of TypeScript: 250983
Nodes of JavaScript: 0
Nodes of JSON: 2256
Nodes of Other: 0
Identifiers: 382980
Symbols: 597111
Types: 216450
Instantiations: 7252666
Memory used: 684086K
Assignability cache size: 202376
Identity cache size: 3213
Subtype cache size: 12303
Strict subtype cache size: 4719
I/O Read time: 0.11s
Parse time: 1.95s
ResolveModule time: 0.58s
ResolveTypeReference time: 0.01s
Program time: 3.04s
Bind time: 1.00s
Check time: 16.47s
printTime time: 0.00s
Emit time: 0.00s
Total time: 20.50s
Here is 4.9
Files: 2980
Lines of Library: 33853
Lines of Definitions: 233669
Lines of TypeScript: 59133
Lines of JavaScript: 0
Lines of JSON: 943
Lines of Other: 0
Identifiers: 384065
Symbols: 571900
Types: 18112640
Instantiations: 7055141
Memory used: 5726426K
Assignability cache size: 10914208
Identity cache size: 3215
Subtype cache size: 11958
Strict subtype cache size: 4719
I/O Read time: 0.15s
Parse time: 1.72s
ResolveModule time: 0.47s
ResolveTypeReference time: 0.01s
Program time: 2.65s
Bind time: 0.79s
Check time: 199.89s
printTime time: 0.00s
Emit time: 0.00s
Total time: 203.33s
I also generated a trace
From looking into the trace. It seems to be stuck in a loop related to the i18next & react-i18next
This is basically the part that loops (if I followed the trace right)
import type { Namespace, TFuncKey } from 'i18next';
import { i18n } from './i18n';
export interface I18nText<N extends Namespace = Namespace> {
namespace: N;
keyName: TFuncKey<N>; // <-- TFuncKey seems to be the culprit
}
That lib is for sure complicated typewise. But it worked as it should in 4.8 and breaks compilation in 4.9 and 5.0
Anything else I can provide? @jakebailey
We've made some improvements recently that feel like this issue; can you try the nightly build?
Using Version "5.1.0-dev.20230328"
Running NODE_OPTIONS=--max-old-space-size=8192 tsc --noEmit --pretty --diagnostics --extendedDiagnostics --incremental false
Result
/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:114282
throw e;
^
RangeError: Value undefined out of range for undefined options property undefined
at Map.set (<anonymous>)
at recursiveTypeRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:60159:24)
at isRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:59590:122)
at eachTypeRelatedToType (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:59946:25)
at unionOrIntersectionRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:59768:174)
at isRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:59590:39)
at structuredTypeRelatedToWorker (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:60559:25)
at structuredTypeRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:60172:21)
at recursiveTypeRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:60142:19)
at isRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:59590:122)
Note
This took about 4mins to run before failing
Uh, hm, well, it's certainly not supposed to crash anymore. If you have the time, if you can roll it back by date to find when it started crashing, that would be helpful.
Phew! Ok finally I got it.
Last version that did not crash. 5.0.0-dev.20221103
Which makes the first version that crashed: 5.0.0-dev.20221108
Note that they were still very slow. And still needed NODE_OPTIONS=--max-old-space-size=8192` set.
Small Note
From
5.0.0-dev.20230210and lower the error message slightly changed from what I posted earlier to
/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:113758
throw e;
^
RangeError: Value undefined out of range for undefined options property undefined
at Map.set (<anonymous>)
at getIndexedAccessTypeOrUndefined (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:56795:28)
at getIndexedAccessType (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:56764:12)
at /home/jarek/Development/project/node_modules/typescript/lib/tsc.js:56687:68
at map (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:209:19)
at distributeIndexOverObjectType (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:56687:21)
at getSimplifiedIndexedAccessType (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:56710:37)
at getSimplifiedType (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:56683:55)
at getNormalizedType (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:58949:481)
at isRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:59262:23)
Oh, yikes, those are the versions before and after the module transform (where things were temporarily broken). Is your project somewhere that I can test it?
It is not. Tomorrow I can perhaps try and create a repo that replicates some of the i18next stuff we are doing with types and see if I can get it to have the same issue.
@jakebailey Well hey it wasn't that bad, I was able to replicate it with little code
https://github.com/JarekToro/TSC-Regression-test-5.0
I understand this to be the offending line. https://github.com/JarekToro/TSC-Regression-test-5.0/blob/main/utils.ts#L15
If you remove that line. Or remove the function in its entirety. It builds correctly.
Running NODE_OPTIONS=--max-old-space-size=8192 tsc --noEmit --pretty --diagnostics --extendedDiagnostics --incremental false To Test
I'm trying to look at this, but it seems like this thread contains many different OOMs and claims about which versions do and don't work.
The only repro I have is @JarekToro's, but that explicitly says that something changed in 5.0. But, if I go back to TS 4.8, that code fails to compile entirely, saying that it's excessively deep. 4.9 hangs, as does 5.0. Is it supposed to complete?
But, that's still not the original reported issue. I guess what I should be checking is https://github.com/Eranziel/foundryvtt-lancer/tree/oom?
Testing the case from the original issue, a bisect points to #51151. @weswigham
Reverting the above also fixes @JarekToro's case, so at least it seems like all of the cases in this thread with repros are the same thing.
My OOM complaint is not a regression per se, just an artifact of a large codebase, I had multiple different versions & copies of the entire aws-sdk v2 library loaded into my project and each copy of aws-sdk has an insanely large amount of symbols.
I don't have a reproduction example at hand, but npm dedupe aws-sdk also resolved my OOM issue to reduce the amount of symbols.
Reverting the above also fixes @JarekToro's case, so at least it seems like all of the cases in this thread with repros are the same thing.
@jakebailey That's good to hear, So yes in regards to my reproduction I was trying to focus on what version the build failed at. Which while related is technically different. Then the original issue. And while I simplified the code base to reproduce the crash. I didn't check to see if it passed on 4.8.4. If I have time I will try to make a separate repro of the slow down difference.
My OOM complaint is not a regression per se, just an artifact of a large codebase, I had multiple different versions & copies of the entire
aws-sdkv2 library loaded into my project and each copy ofaws-sdkhas an insanely large amount of symbols.I don't have a reproduction example at hand, but
npm dedupe aws-sdkalso resolved my OOM issue to reduce the amount of symbols.
Yeah, that might just be a different issue. We've had similar problems on DefinitelyTyped regarding that. If you can see something change between two different TS versions in the --extendedDiagnostics, that might be interesting, though.
Have same issue with 5.0.4 along with 5.1.0-beta
also using i18next along with react-i18next
Have same issue with
5.0.4along with5.1.0-betaalso usingi18nextalong withreact-i18next
Same issue for me, and we are also using i18next and react-i18next.
Facing the same issue with i18next and react-i18next, tested with [email protected] and still no luck 😑 Any timeline on a fix for this issue?
I have a workaround working for [email protected]. I forked off of @JarekToro's repo and implemented it:
https://github.com/wrdecurtins/TSC-4.9.5-workaround
This comment on an i18next issue suggested removing resources from CustomTypeOptions to improve performance during tsc compilation, which seems to work well for [email protected]:
https://github.com/i18next/react-i18next/issues/1600#issuecomment-1374277198
You can also redeclare the TFunction interface to make it less complicated.
declare module 'i18next' {
interface TFunction {
(key: string, options?: {[key: string]: string}): string
}
}
These changes reduce the type safety of the i18next libraries, so I wouldn't recommend using this as anything other than a temporary workaround to get to [email protected].
Unfortunately this doesn't seem to do anything to fix the issue for ts@5+.
There is a pretty good vscode i18next extension that can help restore type safety and autofill i18n keys maintained by the lokalise team:
https://github.com/lokalise/i18n-ally
Also rolling back the i18next package to v21.10.0 and react-i18next to v11.16.9 worked for me.
facing the same problem when upgrading to TS 5+
our tsc build slowly grows its memory usage until it eventually crashes with @whitespine 's error
