OOM crashes in v3.0.0-dev1

[1ma]

Description

The backend component of mempool v3.0.0-dev1 seems to be running out of memory every few hours. I've been running v2.5.0 since March and never saw this error before.

Also, when the backend crashes the mempool-mempool.pid is not cleaned up and prevents the service from starting again.

Version

v3.0.0-dev1

Steps to reproduce

Spin up a mempool backend v3.0.0-dev1 with the manual setup (no docker), let it run for a few hours.

This is my systemd unit:

[Unit]
Description=Mempool Backend
After=network.target bitcoind.service electrs.service

[Service]
Type=exec
WorkingDirectory=/var/www/mempool/backend
ExecStart=/usr/bin/npm run start
User=mempool
Group=mempool

Restart=always
RestartSec=30

[Install]
WantedBy=multi-user.target

Expected behaviour

No crashes

Actual behaviour

<--- Last few GCs --->
[126734:0x76d70b0] 26108747 ms: Scavenge (reduce) 1935.1 (2003.1) -> 1935.0 (2004.1) MB, 13.3 / 0.0 ms  (average mu = 0.574, current mu = 0.595) allocation failure;
[126734:0x76d70b0] 26109808 ms: Mark-sweep (reduce) 2043.0 (2111.6) -> 1989.1 (2083.6) MB, 573.7 / 0.1 ms  (+ 56.1 ms in 416 steps since start of marking, biggest step 0.8 ms, walltime since start of marking 1696 ms) (average mu = 0.727, current mu = 0.83
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb83f50 node::Abort() [node]
 2: 0xa94834  [node]
 3: 0xd647c0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xd64b67 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xf42265  [node]
 6: 0xf5474d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 7: 0xf5743b v8::internal::Heap::AllocateExternalBackingStore(std::function<void* (unsigned long)> const&, unsigned long) [node]
 8: 0x107bc42 v8::internal::BackingStore::Allocate(v8::internal::Isolate*, unsigned long, v8::internal::SharedFlag, v8::internal::InitializedFlag) [node]
 9: 0xf23ddb v8::internal::Factory::NewJSArrayBufferAndBackingStore(unsigned long, v8::internal::InitializedFlag, v8::internal::AllocationType) [node]
10: 0xd757f7 v8::ArrayBuffer::New(v8::Isolate*, unsigned long) [node]
11: 0xb9d6a5 node::fs::FSReqPromise<node::AliasedBufferBase<double, v8::Float64Array, void> >::New(node::fs::BindingData*, bool) [node]
12: 0xb95ae4  [node]
13: 0xdc28b0  [node]
14: 0xdc3def v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [node]
15: 0x17036b9  [node]
Aborted (core dumped)
mempool.service: Main process exited, code=exited, status=134/n/a
mempool.service: Failed with result 'exit-code'.
mempool.service: Consumed 6h 31min 48.117s CPU time.
mempool.service: Scheduled restart job, restart counter is at 507.
Stopped Mempool Backend.
mempool.service: Consumed 6h 31min 48.117s CPU time.
Starting Mempool Backend...
Started Mempool Backend.
> [email protected] start
> node --max-old-space-size=2048 dist/index.js
Nov 18 04:33:49 [135373] NOTICE: Starting Mempool Server... (79fb0a3)
Nov 18 04:33:49 [135373] ERR: Already running on PID 126734 (or pid file '/var/www/mempool/backend/dist/mempool-mempool.pid' is stale)
mempool.service: Deactivated successfully.
mempool.service: Consumed 1.521s CPU time.
mempool.service: Scheduled restart job, restart counter is at 508.
Stopped Mempool Backend.
mempool.service: Consumed 1.521s CPU time.
Starting Mempool Backend...
Started Mempool Backend.
> [email protected] start
> node --max-old-space-size=2048 dist/index.js
Nov 18 04:35:54 [135494] NOTICE: Starting Mempool Server... (79fb0a3)
Nov 18 04:35:54 [135494] ERR: Already running on PID 126734 (or pid file '/var/www/mempool/backend/dist/mempool-mempool.pid' is stale)
mempool.service: Deactivated successfully.
mempool.service: Consumed 1.376s CPU time.
mempool.service: Scheduled restart job, restart counter is at 512.
Stopped Mempool Backend.
mempool.service: Consumed 1.376s CPU time.
Starting Mempool Backend...

infinite restart loop...

Device or machine

amd64 box with 16 GB of RAM, running Ubuntu Server 22.04.3

Additional info

NodeJS v18.17.1 npm 9.6.7

[mononaut]

By default, nodejs refuses to use more than ~2GB of heap space. That should be enough for a default mempool, but you'll need to increase the limit if you're running a node with higher maxmempool settings.

Either:

• run with the production start command npm run start-production (sets the heap limit to 16GB).
• start the backend with node --max-old-space-size=4096 dist/index.js for a custom limit.
• set via the NODE_OPTIONS environment variable: export NODE_OPTIONS=--max_old_space_size=4096.

[mononaut]

The pid file issue should already be resolved by https://github.com/mempool/mempool/pull/4356

[1ma]

I'm running with a double node configuration using a 4GB mempool and the secondary with the regular one, should've mentioned it.

I've manually set the max-old-space-size flag to 4096 and I'll wait to see if the errors persist. If they don't I'll wait for the next tagged release, if not I'll target that commit with the PID fix.

Thanks for your help.