indexer: "JavaScript out of memory" fatal errors

[adrianopol]

Hello, I regularly (every 1--2 days) get the following error

2021-01-24T02:55:27: FATAL ERROR: invalid table size Allocation failed - JavaScript heap out of memory
2021-01-24T02:55:27:  1: 0xa0e670 node::Abort() [hyp-daobet-master]
2021-01-24T02:55:27:  2: 0xa0ea9c node::OnFatalError(char const*, char const*) [hyp-daobet-master]
2021-01-24T02:55:27:  3: 0xb83afe v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [hyp-daobet-master]
2021-01-24T02:55:27:  4: 0xb83e79 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [hyp-daobet-master]
2021-01-24T02:55:27:  5: 0xd32305  [hyp-daobet-master]
2021-01-24T02:55:27:  6: 0xf23cba v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::New(v8::internal::Isolate*, int, v8::internal::AllocationType, v8::internal::MinimumCapacity) [hyp-daobet-master]
2021-01-24T02:55:27:  7: 0xf2437b v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::EnsureCapacity(v8::internal::Isolate*, v8::internal::Handle<v8::internal::StringTable>, int, v8::internal::AllocationType) [hyp-daobet-master]
2021-01-24T02:55:27:  8: 0xf2d0d1 v8::internal::Handle<v8::internal::String> v8::internal::StringTable::LookupKey<v8::internal::InternalizedStringKey>(v8::internal::Isolate*, v8::internal::InternalizedStringKey*) [hyp-daobet-master]
2021-01-24T02:55:27:  9: 0xf2d1c6 v8::internal::StringTable::LookupString(v8::internal::Isolate*, v8::internal::Handle<v8::internal::String>) [hyp-daobet-master]
2021-01-24T02:55:27: 10: 0x107306b v8::internal::Runtime_GetProperty(int, unsigned long*, v8::internal::Isolate*) [hyp-daobet-master]
2021-01-24T02:55:27: 11: 0x13da519  [hyp-daobet-master]

OS: Ubuntu 18.04 amd64. Server: 64 GiB RAM, 8 cores. Installation type: no Docker.

Chain config:

{
  "api": {
    ...
    "enable_caching": true,
    "cache_life": 1,
    "limits": {
      "get_actions": 1000,
      "get_voters": 100,
      "get_links": 1000,
      "get_deltas": 1000,
      "get_trx_actions": 200
    },
    "access_log": false,
    "enable_explorer": false,
    "chain_api_error_log": false
  },
  "settings": {
    "preview": false,
    ...
    "parser": "1.8",
    "auto_stop": 300,
    "index_version": "v1",
    "debug": false,
    "bp_logs": false,
    "bp_monitoring": true,
    "ipc_debug_rate": 60000,
    "allow_custom_abi": false,
    "rate_monitoring": true,
    "max_ws_payload_kb": 256,
    "ds_profiling": false,
    "auto_mode_switch": false
  },
  "blacklists": {
    "actions": [],
    "deltas": []
  },
  "whitelists": {
    "actions": [],
    "deltas": [],
    "max_depth": 10,
    "root_only": false
  },
  "scaling": {
    "readers":            2,
    "ds_queues":          2,
    "ds_threads":         2,
    "ds_pool_size":       2,
    "indexing_queues":    2,
    "ad_idx_queues":      1,
    "max_autoscale":      4,
    "batch_size":         5000,
    "resume_trigger":     5000,
    "auto_scale_trigger": 20000,
    "block_queue_limit":  10000,
    "max_queue_limit":    100000,
    "routing_mode":       "heatmap",
    "polling_interval":   10000
  },
  "indexer": {
    "start_on": 0,
    "stop_on": 0,
    "rewrite": false,
    "purge_queues": false,
    "live_reader": true,
    "live_only_mode": false,
    "abi_scan_mode": false,
    "fetch_block": true,
    "fetch_traces": true,
    "disable_reading": false,
    "disable_indexing": false,
    "process_deltas": true
  },
  "features": {
    "streaming": {
      "enable": false,
      "traces": false,
      "deltas": false
    },
    "tables": {
      "proposals": true,
      "accounts": true,
      "voters": true,
      "userres": false,
      "delband": false
    },
    "index_deltas": false,
    "index_transfer_memo": false,
    "index_all_deltas": true,
    "deferred_trx": false,
    "failed_trx": false,
    "resource_limits": false,
    "resource_usage": false
  },
  "prefetch": {
    "read": 50,
    "block": 100,
    "index": 500
  }
}

[bublik77]

Hi, probably you should increase "heap size", the default value is 4096 file is in definitions/ecosystem_settings.js value "--max-old-space-size"

[adrianopol]

Thanks, I will wait for a couple of days to see, how it feels.

[adrianopol]

Yes, looks like it helped.

[adrianopol]

I increased max-old-space-size up to 32 MiB, hyperion was OK for some time, but the last weekends I got another memory-exhaustion problem. After restart, indexer worked from 30 minutes to 2 hours, breaking with the following error:

2021-02-08T21:00:11: [00_master] --------- Hyperion Indexer 3.1.2 ---------
2021-02-08T21:00:11: [00_master] Using parser version 1.8
2021-02-08T21:00:11: [00_master] Chain: daobet
...
2021-02-08T15:14:29: [03_continuous_reader] Removing indexed data from 72002918 to 72002918
2021-02-08T15:14:29: [03_continuous_reader] Live reading resumed!
2021-02-08T15:16:44: 
2021-02-08T15:16:44: <--- Last few GCs --->
2021-02-08T15:16:44: 
2021-02-08T15:16:44: [28889:0x4567e70]  4691330 ms: Mark-sweep 32766.9 (32805.6) -> 32766.6 (32805.9) MB, 24972.4 / 0.0 ms  (average mu = 0.097, current mu = 0.000) allocation failure scavenge might not succeed
2021-02-08T15:16:44: [28889:0x4567e70]  4718705 ms: Mark-sweep 32767.2 (32805.9) -> 32767.0 (32806.1) MB, 27123.5 / 0.0 ms  (+ 192.8 ms in 10 steps since start of marking, biggest step 165.7 ms, walltime since start of marking 27374 ms) (average mu = 0.050, current mu = 0.002
2021-02-08T15:16:44: 
2021-02-08T15:16:44: <--- JS stacktrace --->
2021-02-08T15:16:44: 
2021-02-08T15:16:44: ==== JS stack trace =========================================
2021-02-08T15:16:44: 
2021-02-08T15:16:44:     0: ExitFrame [pc: 0x13da519]
2021-02-08T15:16:44:     1: StubFrame [pc: 0x13db355]
2021-02-08T15:16:44: Security context: 0x2c790c600921 <JSObject>
2021-02-08T15:16:44:     2: onLiveBlock [0x597ec274989] [/root/hyperion-history-api/modules/master.js:~731] [pc=0x1594c0e44b66](this=0x15dc03dcae21 <HyperionMaster map = 0xbacf3454f81>,0x38b2845b1989 <Object map = 0xbacf3454531>)
2021-02-08T15:16:44:     3: process [0x15dc03dcbc11] [/root/hyperion-history-api/node_modules/async/dist/async.js:~1398] [pc=0x1594c0e10cdd](thi...
2021-02-08T15:16:44: 
2021-02-08T15:17:11: [03_continuous_reader] Expected: 72003230, received: 72003229

There were frequent blockchain forks that time, maybe they triggered this problem.

---

Branch: master; commit: 28c163a5000e8ae22a57a17053efffd2593fb19f.

[igorls]

Thanks, we are investigating those issues