(reth) high memory usage after running for a long period and high usage

[calebcall]

trafficstars

Describe the bug

Seems to be a memory leak in reth after running for a longer period of time. This isn't during an initial sync, node is otherwise keeping up with the tip of the chain. Currently observing similar behavior across several nodes all configured identical.

Attached screenshot showing cadence of OOMKiller events that happen over this timeframe:

Each time reth is killed it's up around 120Gb of memory used

Steps to reproduce

1. Start node
2. run node for period of time with decent usage (roughly 100rps)
3. high memory usage ensues

Node logs

Platform(s)

Linux (x86)

What version/commit are you on?

# reth --version
reth Version: 1.0.6
Commit SHA: c228fe15808c3acbf18dc3af1a03ef5cbdcda07a
Build Timestamp: 2024-08-29T18:18:33.351492336Z
Build Features: asm_keccak,jemalloc
Build Profile: maxperf

# lighthouse --version
Lighthouse v5.1.3-3058b96
BLS library: blst-modern
SHA256 hardware acceleration: true
Allocator: jemalloc
Profile: maxperf
Specs: mainnet (true), minimal (false), gnosis (true)

Hardware node is running on: 7950X 128G memory NVME4 drive for storage

Ubuntu 22.04

What database version are you on?

# reth db --datadir /blockchain_data/reth version
2024-11-28T19:47:01.516066Z  INFO reth::cli: Initialized tracing, debug log directory: /root/.cache/reth/logs/mainnet
Current database version: 2
Local database version: 2

Which chain / network are you on?

mainnet

What type of node are you running?

Full via --full flag

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• [x] I agree to follow the Code of Conduct