FATAL ERROR: NewSpace::Rebalance Allocation failed - JavaScript heap out of memory

[drouarb]

• Node.js Version: v14.2.0 (also tried in v13.8.0)
• OS: Ubuntu 18.04 Server
• Scope (install, code, runtime, meta, other?): Runtime
• Module (and version) (if relevant): Typescript, sequelize, node-pg (all lastest)

I'm running a script, doing heavy data-processing in worker-threads, it runs until node crash with the following error:

FATAL ERROR: NewSpace::Rebalance Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20200512.172528.47517.24.011.json
Node.js report completed
 1: 0xa295e0 node::Abort() [node]
 2: 0x9782df node::FatalError(char const*, char const*) [node]
 3: 0xb99c2e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb99fa7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd3a3b5  [node]
 6: 0xd74f27  [node]
 7: 0xd84707 v8::internal::MarkCompactCollector::CollectGarbage() [node]
 8: 0xd481b9 v8::internal::Heap::MarkCompact() [node]
 9: 0xd48f0b v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
10: 0xd499a5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
11: 0xd4aebf v8::internal::Heap::HandleGCRequest() [node]
12: 0xcf5f97 v8::internal::StackGuard::HandleInterrupts() [node]
13: 0x104b803 v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [node]
14: 0x13a5a99  [node]
Aborted (core dumped)

This error appears randomly after few hours of processing. The worker are on the following architecture:

Main thread (idle) => Coordination Thread (Dispatch Job) => Worker Thread
                                                         => Worker Thread
                                                         => Worker Thread
                                                         => ... (32x)

Here is the end of my log: https://gist.github.com/drouarb/28f3b411a3088bc85bf82e65d0def25d

>>>>> Task Memory
>>>>> rss: 19209M
>>>>> heapTotal: 80M
>>>>> heapUsed: 78M
>>>>> external: 5M
>>>>> arrayBuffers: 4M

Is a log I added in a setInterval in the main thread.

==================================================
rss: 18572M
heapTotal: 758M
heapUsed: 725M
external: 196M
arrayBuffers: 195M
=====================After GC=====================
rss: 18628M
heapTotal: 274M
heapUsed: 88M
external: 196M
arrayBuffers: 195M
==================================================

Is a log of the memory before and after GC in Workers thread.

I added the following flags to node: --expose-gc --report-on-fatalerror --report-uncaught-exception --report-on-signal, that allow me to have a report once node crash.

Here is a diagnostic generated with a SIGUSR2 while running: https://gist.github.com/drouarb/2f919f5152eb44dc569ea1ce6bf4accf

Here is the diagnostic when node crashed: https://gist.github.com/drouarb/875ac90e074fca3b9b029be5633f1e09 I Converted all sizes to human size.

It appears that my workers disappeared when node generated the last crash.

My workers thread are sending data to the coordination thread with ArrayBuffer which are transfered, could it be a source of the memory leak ?

On crash we see that old_space memorySize have grown, but not the capacity. Also new_space=>memorySize jumped form 1.05 MB to 33.6 MB, is it normal ? The main thread, there is nothing happening except the display of a process.memoryUsage()

If it is a worker going out of memory, why don't I have a ERR_WORKER_OUT_OF_MEMORY ?

EDIT 1: After looking at the report, it appears that the log comes from thread 24 ("threadId": 24 in the Diagnostic Report no 2). EDIT 2: I'm running my app again, and logging every 5seconds node Diagnostic Reports to grafana. I hope I will find something intersting.