Polygon Amoy nodes memory spikes

[jonathanudd]

System information

Bor client version: 2.5.1

Heimdall client version: 0.5.0-beta

OS & Version: Ubuntu 22.04

Environment: Polygon Amoy

Type of node: Archive

Overview of the problem

Since around 2025-11-19 19:30 our Bor running Polygon Amoy started having memory spikes making the process running out of memory. Our nodes have had 32GB available for a long time without issues and even after increasing available memory to 48GB they still run out of memory when these spikes occur.

Reproduction Steps

Please mention the steps required to reproduce this issue.

1. Run a node using these flags

--chain amoy --syncmode full --gcmode archive --http --http.addr 0.0.0.0 --http.port 8545 --http.vhosts * --http.api eth,net,web3,txpool,bor,debug --metrics --metrics.prometheus-addr 0.0.0.0:9101 --maxpeers 350 --maxpendpeers 200 --txlookuplimit 0 --port=30374 --nat=extip:79.136.114.244 --discovery.dns enrtree://AKUEZKN7PSKVNR65FZDHECMKOJQSGPARGTPPBI7WS2VUL4EGR6XPC@amoy.polygon-peers.io --db.engine=leveldb --state.scheme=hash --bootnodes=enode://0ef8758cafc0063405f3f31fe22f2a3b566aa871bd7cd405e35954ec8aa7237c21e1ccc1f65f1b6099ab36db029362bc2fecf001a771b3d9803bbf1968508cef@35.197.249.21:30303,enode://c9c8c18cde48b41d46ced0c564496aef721a9b58f8724025a0b1f3f26f1b826f31786f890f8f8781e18b16dbb3c7bff805c7304d1273ac11630ed25a3f0dc41c@34.89.39.114:30303,enode://5b8d436677fb545b1c3fd1ae84553d478d9d21ad3b06a908b9d34d2df367ead5bb8823d84a370e26bdde8896ba8a870e21ba3a6dce19c0ded086296df5f04f15@35.242.167.175:30303,enode://5bd810da4f021a974e80cf2be48975d58cafbcfdd971d568ab98250f8568a9457bdc1b7a6d16b5aebfcb9deb0c1ec612f0664d5366c74f5266906a2774dd70f0@34.89.15.223:30303,enode://a2ec3671e553ba3e711639033912be55fe1e7fa4b61a93f6a1ac0cd3cea34f9d7eec1d718e04049531cf5dd7efc1ac677df1cf0e1f24f5e677706d7bcb3917de@34.105.128.110:30303,enode://9e15bc58779c32119140d54a8384940b57a10a001506ce173cc4cdb10876b14a2ac9ae91f9389caf9fd385c3b72825f8bbbe937e7e57b1f032561703e900da59@34.89.21.99:30303,enode://42203e9b423aba24e1e9386f94d0d0397a42770427e8e9e22f9e2a9523f66abb13b1f5a6addee68ad5986f94a8f6de626f5829492599a2f9484f98e86e26149d@34.89.101.16:30303,enode://83c235bb4305ecdd5addcbfd09478d2df7cddca9b7eed3ef11b2426fad04ccfe7335279e2371a85696e461dcbe78de6912d07043a912dbd85cb0bb944d78b8d9@34.89.11.233:30303 --ws --ws.api eth,net,web3,txpool,bor,debug --ws.addr=0.0.0.0 --ws.port=8546

2. The node will occasionally have memory spikes resulting in the Bor service being killed

Logs / Traces / Output / Error Messages

bor-amoy.log

[marcello33]

This apparently was identified by other operators as well, but most probably related to network activity and load testing. @jonathanudd can you confirm this is no longer happening for you as well? Thanks

[jonathanudd]

This apparently was identified by other operators as well, but most probably related to network activity and load testing. @jonathanudd can you confirm this is no longer happening for you as well? Thanks

Our nodes is syncing and working properly but I do still see increased resource usage from the nodes since this event without increased RPS load. The resource usage is quite spiky.

[ShisheerBitgo]

@marcello33 Out of curiosity, are there any updates on this testnet (Amoy) issue?

I have been observing the same behaviour with my nodes running Heimdall v0.5.2 and Bor v2.5.4. I briefly gave my nodes over 300Gi` memory and it ate it up.

More details on my Bor arguments in this issue

A good example of this is:

{
    "jsonrpc": "2.0",
    "method": "debug_traceBlockByNumber",
    "params": [
        "0x1BE7FC0",
        {
            "tracer": "callTracer",
            "borTraceEnabled": true
        }
    ],
    "id": 1
}

This block has a lot of transactions (think the response is over 700MB) and makes Bor memory go crazy. For blocks that do not have too many transactions, the memory consumption seems fine

[github-actions[bot]]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.