OOMKilled with sudden memory spike

[erwin-wee]

Describe the bug

While running a 15 minute load test on reth with a load profile consisting of rpc calls from debug and trace namespaces, there is a sudden memory spike and the client gets OOMKilled.

Load Profile

Percentage	Method
61.42%	debug_traceTransaction
24.14%	trace_transaction
7.11%	trace_block
3.00%	debug_traceCall
1.58%	trace_replayTransaction
1.25%	trace_callMany
0.86%	trace_filter
0.30%	debug_traceBlockByHash
0.29%	debug_traceBlockByNumber
0.05%	debug_getRawReceipts

Cli flags:

exec /usr/local/bin/reth node --datadir=/data --config=/data/config.toml --nat=extip:$(POD_IP) --port=30303 --http --http.api all --http.addr=0.0.0.0 --http.port=8545 --http.corsdomain=* --ws --ws.api all --ws.addr=0.0.0.0 --ws.port=8546 --ws.origins=* --authrpc.jwtsecret=/data/jwt.hex --authrpc.addr=0.0.0.0 --authrpc.port=8551 --log.file.directory=/data/logs --metrics=0.0.0.0:9000

Docker resources:

        resources:
          limits:
            cpu: 7500m
            memory: 28Gi
          requests:
            cpu: 7500m
            memory: 28Gi

Memory Metrics:

RPC Server Metrics:

Full Grafana Screen:

Steps to reproduce

1. Run Load Test
2. Observe and monitor

Node logs

No response

Platform(s)

Linux (x86)

What version/commit are you on?

reth Version: 1.0.0 Commit SHA: 83d412d Build Timestamp: 2024-06-24T16:45:09.014806662Z Build Features: jemalloc Build Profile: maxperf

What database version are you on?

Getting an error message when trying to get db version

thread 'main' panicked at crates/tracing/src/layers.rs:136:46:
Could not create log directory: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }

Which chain / network are you on?

Ethereum Mainnet

What type of node are you running?

Archive (default)

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• [X] I agree to follow the Code of Conduct