Tree retains memory in global? scope - causes OOM :warning:

[dapplion]

This gist demostrates a case where memory is retained in some upper scope such that it's not garbage collected.

https://gist.github.com/dapplion/e079d8faf795f758f1f3b341ee590dc2

import {ssz} from "@chainsafe/lodestar-types";

let i = 0;
const heapUsed = process.memoryUsage().heapUsed;

while (true) {
  getBigStateTreeBacked();
  global.gc();
  console.log(i++, (process.memoryUsage().heapUsed - heapUsed) / 1e6, "MB");
}

function getBigStateTreeBacked(): any {
  const stateTB = ssz.phase0.BeaconState.defaultTreeBacked();
  const validator = ssz.phase0.Validator.defaultValue();
  for (let i = 0; i < 250_000; i++) {
    stateTB.validators.push(validator);
  }
}

lodestar$ LODESTAR_PRESET=mainnet node --expose-gc ts-node lodestar-ssz-oom.ts
0 456.3956 MB
1 919.7116 MB
2 1388.220224 MB
3 1846.017496 MB

<--- Last few GCs --->

[26583:0x5ab6bc0]    37176 ms: Mark-sweep (reduce) 2046.8 (2053.8) -> 2045.9 (2053.8) MB, 1838.1 / 0.0 ms  (+ 0.0 ms in 29 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1845 ms) (average mu = 0.150, current mu = 0.016)[26583:0x5ab6bc0]    38694 ms: Mark-sweep (reduce) 2047.0 (2050.8) -> 2046.1 (2052.0) MB, 1515.4 / 0.0 ms  (average mu = 0.081, current mu = 0.001) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xa25510 node::Abort() [node]
 2: 0x9664d3 node::FatalError(char const*, char const*) [node]
 3: 0xb9a8be v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb9ac37 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd56ca5  [node]
 6: 0xd5782f  [node]
 7: 0xd6566b v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 8: 0xd6922c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 9: 0xd3790b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
10: 0x107fbef v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x1426919  [node]
Aborted (core dumped)

[dapplion]

The cause has been traced back to the SubTree WeakRef

• nodejs/node issue (closed) https://github.com/nodejs/node/issues/39902
• v8 issue https://bugs.chromium.org/p/v8/issues/detail?id=12145

[twoeths]

we switched to ssz v2 a long time ago, closing the issue