erigon icon indicating copy to clipboard operation
erigon copied to clipboard
verified publisher icon erigontech

Erigon v2.60.8 problem during automatic unwind

Open MrFreezeDZ opened this issue 1 year ago • 3 comments

System information

Erigon version: ./erigon --version We run the following image here: thorax/erigon:2.60.8

OS & Version: Windows/Linux/OSX GKE Kubernetes running the Image thorax/erigon:2.60.8

Commit hash: {"git_branch":"heads/2.60.8", "git_commit":"dec9d2067be63cb155016efa05050339e2a8994a", "git_tag":"v2.60.6-32-gdec9d2067b-dirty", "lvl":"info", "msg":"Build info", "t":"2024-12-08T04:41:18.639638773Z"}

Erigon Command (with flags/config):

    spec:
      containers:
      - args:
        - --chain=bor-mainnet
        - --datadir=/data/
        - --log.json=true
        - --http.addr=0.0.0.0
        - --http.vhosts=*
        - --http
        - --http.api=eth,debug,trace
        - --ws
        - --bor.heimdall=http://heimdallrest
        - --authrpc.vhosts=*
        - --authrpc.jwtsecret=/secret/jwt.hex
        - --authrpc.addr=0.0.0.0
        - --db.size.limit=7TB
        - --db.pagesize=16k
        - --metrics
        - --metrics.addr=0.0.0.0
        - --prune.h.before=48896162
        - --prune.r.before=48896162
        - --prune.t.before=48896162
        - --prune.c.before=48896162
        - --maxpeers=500
        - --torrent.download.rate=300mb
        command:
        - erigon
        env:
        - name: SAVE_HEAP_PROFILE
          value: "true"

Consensus Layer: Heimdall

Consensus Layer Command (with flags/config): Heimdall Image: 0xpolygon/heimdall:1.0.7

Chain/Network: Polygon/Bor-Mainnet

Expected behaviour

If Erigon needs to unwind automatically it will do so and keep working.

Actual behaviour

Erigon tried to unwind automatically and failed, I guess. Since then it logs the same every millisecond.

Steps to reproduce the behaviour

I do not know, it happened on its own.

Backtrace

The beginning ERROR on each line with the timestamp is from the LogsExplorer, the json logs are coming from erigon. I think the most interesting lines are around the one beginning with "ERROR 2024-12-17T18:06:58.175630096Z" Logs around the first occurance from the Google LogsExplorer:

ERROR 2024-12-17T18:05:09.779520105Z {"from":6.5601811E7, "lvl":"info", "msg":"[2/15 Headers] Waiting for headers...", "t":"2024-12-17T18:05:09.779221355Z"}
ERROR 2024-12-17T18:05:09.887949798Z {"age":"0", "blk/sec":110, "headers":12, "highest":6.5601823E7, "in":0.108475693, "lvl":"info", "msg":"[2/15 Headers] Processed", "t":"2024-12-17T18:05:09.887680838Z"}
ERROR 2024-12-17T18:05:09.887998428Z {"from":6.5601812E7, "lvl":"info", "msg":"[3/15 BorHeimdall] Processing sync events...", "t":"2024-12-17T18:05:09.887822898Z", "to":6.5601823E7}
ERROR 2024-12-17T18:05:09.947890255Z {"fetch time":"0s", "lastCheckpointId":0, "lastMilestoneId":0, "lastSpanID":10251, "lastStateSyncEventID":2997706, "lvl":"info", "msg":"[3/15 BorHeimdall] Sync events processed", "process time":"59.858807ms", "progress":6.5601823E7, "snap time":"59.464744ms", "sync event time":"29.72329ms", "t":"2024-12-17T18:05:09.947637735Z", "total records":0, "waypoint time":"0s"}
ERROR 2024-12-17T18:05:15.748970802Z {"block":6.5601823E7, "lvl":"info", "msg":"[7/15 Execution] Completed on", "t":"2024-12-17T18:05:15.748630412Z"}
ERROR 2024-12-17T18:05:46.372978257Z {"amount":12, "from":6.5601811E7, "hash":"0x85afb253a5f770643ffeddcf964f8743a0fa2e037716654db407502ac68bf7ed", "header sending":"62.44µs", "log sending":"120ns", "lvl":"info", "msg":"RPC Daemon notified of new headers", "t":"2024-12-17T18:05:46.372673897Z", "to":6.5601823E7}
ERROR 2024-12-17T18:05:46.373031107Z {"in":"1.305416068s", "lvl":"info", "msg":"Commit cycle", "t":"2024-12-17T18:05:46.372862157Z"}
ERROR 2024-12-17T18:05:46.373054967Z {"AccountHistoryIndex":"598ms", "BorHeimdall":"59ms", "CallTraces":"1.992s", "Execution":"5.728s", "HashState":"6.038s", "Headers":"108ms", "IntermediateHashes":"12.282s", "LogIndex":"3.855s", "StorageHistoryIndex":"3.242s", "TxLookup":"1.308s", "lvl":"info", "msg":"Timings (slower than 50ms)", "t":"2024-12-17T18:05:46.372892687Z"}
ERROR 2024-12-17T18:05:46.373076307Z {"AccountChangeSet":"152.1GB", "BlockTransaction":"357.5GB", "FreeList":"46.3MB", "PlainState":"481.4GB", "ReclaimableSpace":"185.4GB", "StorageChangeSet":"463.3GB", "TransactionLog":"2.2TB", "lvl":"info", "msg":"Tables", "t":"2024-12-17T18:05:46.372927207Z"}
ERROR 2024-12-17T18:05:47.890554082Z {"from":6.5601823E7, "lvl":"info", "msg":"[2/15 Headers] Waiting for headers...", "t":"2024-12-17T18:05:47.890359592Z"}
ERROR 2024-12-17T18:05:47.999508255Z {"age":"0", "blk/sec":82, "headers":9, "highest":6.5601832E7, "in":0.108918113, "lvl":"info", "msg":"[2/15 Headers] Processed", "t":"2024-12-17T18:05:47.999263225Z"}
ERROR 2024-12-17T18:05:47.999557615Z {"from":6.5601824E7, "lvl":"info", "msg":"[3/15 BorHeimdall] Processing sync events...", "t":"2024-12-17T18:05:47.999453225Z", "to":6.5601832E7}
ERROR 2024-12-17T18:05:48.054905982Z {"fetch time":"3.43704ms", "lastCheckpointId":0, "lastMilestoneId":0, "lastSpanID":10251, "lastStateSyncEventID":2997706, "lvl":"info", "msg":"[3/15 BorHeimdall] Sync events processed", "process time":"55.315657ms", "progress":6.5601832E7, "snap time":"62.961604ms", "sync event time":"23.57854ms", "t":"2024-12-17T18:05:48.054728622Z", "total records":0, "waypoint time":"0s"}
ERROR 2024-12-17T18:05:54.366605767Z {"block":6.5601832E7, "lvl":"info", "msg":"[7/15 Execution] Completed on", "t":"2024-12-17T18:05:54.366271367Z"}
ERROR 2024-12-17T18:06:01.368074810Z {"end":6.5600165E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0xc14e7eb0f5e251d55925335df400af6f2a8e1385b11ad28c3a19f0fb4536fbdf", "start":6.559863E7, "t":"2024-12-17T18:06:01.367835Z"}
ERROR 2024-12-17T18:06:25.394771856Z {"amount":9, "from":6.5601823E7, "hash":"0x1d060161f6797ff3ad93a8ccbe69be0d953e83bc728fc5309b6c3e4dda0a588e", "header sending":"65.889µs", "log sending":"389ns", "lvl":"info", "msg":"RPC Daemon notified of new headers", "t":"2024-12-17T18:06:25.394451576Z", "to":6.5601832E7}
ERROR 2024-12-17T18:06:25.394848896Z {"in":"1.199141734s", "lvl":"info", "msg":"Commit cycle", "t":"2024-12-17T18:06:25.394689236Z"}
ERROR 2024-12-17T18:06:25.394856936Z {"AccountHistoryIndex":"760ms", "BorHeimdall":"55ms", "CallTraces":"1.024s", "Execution":"6.202s", "HashState":"6.203s", "Headers":"109ms", "IntermediateHashes":"10.316s", "LogIndex":"4.406s", "StorageHistoryIndex":"4.102s", "TxLookup":"3.014s", "lvl":"info", "msg":"Timings (slower than 50ms)", "t":"2024-12-17T18:06:25.394735156Z"}
ERROR 2024-12-17T18:06:25.394959347Z {"AccountChangeSet":"152.1GB", "BlockTransaction":"357.5GB", "FreeList":"46.3MB", "PlainState":"481.4GB", "ReclaimableSpace":"185.4GB", "StorageChangeSet":"463.3GB", "TransactionLog":"2.2TB", "lvl":"info", "msg":"Tables", "t":"2024-12-17T18:06:25.394808256Z"}
ERROR 2024-12-17T18:06:26.225835575Z {"from":6.5601832E7, "lvl":"info", "msg":"[2/15 Headers] Waiting for headers...", "t":"2024-12-17T18:06:26.225528375Z"}
ERROR 2024-12-17T18:06:26.446336091Z {"age":"0", "blk/sec":63, "headers":14, "highest":6.5601846E7, "in":0.220573957, "lvl":"info", "msg":"[2/15 Headers] Processed", "t":"2024-12-17T18:06:26.446089801Z"}
ERROR 2024-12-17T18:06:27.901777150Z {"eth67":"4", "eth68":"2", "lvl":"info", "msg":"[p2p] GoodPeers", "t":"2024-12-17T18:06:27.90149864Z"}
ERROR 2024-12-17T18:06:37.243810195Z {"Anonymous":"9.3GB", "PrivateClean":"181.1GB", "PrivateDirty":"9.3GB", "Pss":"190.4GB", "Referenced":"72.5GB", "Rss":"190.4GB", "SharedClean":"4.0KB", "SharedDirty":"0B", "Size":"0B", "Swap":"0B", "alloc":"5.3GB", "lvl":"info", "msg":"[mem] memory stats", "sys":"12.1GB", "t":"2024-12-17T18:06:37.243518026Z"}
ERROR 2024-12-17T18:06:39.960800645Z {"codes":true, "from":6.5601832E7, "lvl":"info", "msg":"[8/15 HashState] Unwinding started", "storage":false, "t":"2024-12-17T18:06:39.960400865Z", "to":6.5601824E7}
ERROR 2024-12-17T18:06:39.966656295Z {"codes":false, "from":6.5601832E7, "lvl":"info", "msg":"[8/15 HashState] Unwinding started", "storage":false, "t":"2024-12-17T18:06:39.966470555Z", "to":6.5601824E7}
ERROR 2024-12-17T18:06:40.188731221Z {"codes":false, "from":6.5601832E7, "lvl":"info", "msg":"[8/15 HashState] Unwinding started", "storage":true, "t":"2024-12-17T18:06:40.188457281Z", "to":6.5601824E7}
ERROR 2024-12-17T18:06:40.731331607Z {"csbucket":"AccountChangeSet", "from":6.5601832E7, "lvl":"info", "msg":"[9/15 IntermediateHashes] Unwinding", "t":"2024-12-17T18:06:40.731030887Z", "to":6.5601824E7}
ERROR 2024-12-17T18:06:40.748847406Z {"csbucket":"StorageChangeSet", "from":6.5601832E7, "lvl":"info", "msg":"[9/15 IntermediateHashes] Unwinding", "t":"2024-12-17T18:06:40.748619856Z", "to":6.5601824E7}
ERROR 2024-12-17T18:06:41.258753754Z {"alloc":"5.6GB", "baseFee":0, "lvl":"info", "msg":"[txpool] stat", "pending":1421, "queued":4790, "sys":"12.1GB", "t":"2024-12-17T18:06:41.258450454Z"}
ERROR 2024-12-17T18:06:41.262622823Z {"hash":"0xbae204cce74a455f49480ec87c99dab1a69404edfc2529134bc4400167a71df6", "lvl":"info", "msg":"[9/15 IntermediateHashes] Trie root", "t":"2024-12-17T18:06:41.262430034Z"}
ERROR 2024-12-17T18:06:41.336745539Z {"expected":"b4226bda6922457130232720b003c1887957acce4d5032e60ccc00431e2e293e", "got":"055fe10231d364206cdc9284080573b5f8e8ef43be9884c209380c6744c0fdbb", "lvl":"warn", "msg":"[bor] End block hash mismatch while whitelisting milestone", "t":"2024-12-17T18:06:41.335709259Z"}
ERROR 2024-12-17T18:06:41.336773830Z {"lvl":"warn", "msg":"[bor] Rewinding chain due to milestone endblock hash mismatch", "number":6.5601807E7, "t":"2024-12-17T18:06:41.33658429Z"}
ERROR 2024-12-17T18:06:42.698982214Z {"from":6.5601832E7, "lvl":"info", "msg":"[7/15 Execution] Unwind Execution", "t":"2024-12-17T18:06:42.698759494Z", "to":6.5601824E7}
ERROR 2024-12-17T18:06:43.603896188Z {"err":"[3/15 BorHeimdall] verification failed for header 65601846: b130a4113ae60482a42c9948c7884c3deda2f7ddc47600031e3c6146bc9b80b1", "lvl":"eror", "msg":"Staged Sync", "t":"2024-12-17T18:06:43.603626218Z"}
ERROR 2024-12-17T18:06:52.002009732Z {"from":6.5601832E7, "lvl":"info", "msg":"[10/15 CallTraces] Unwind", "t":"2024-12-17T18:06:52.001695872Z", "to":6.5601807E7}
ERROR 2024-12-17T18:06:53.167171259Z {"codes":true, "from":6.5601832E7, "lvl":"info", "msg":"[8/15 HashState] Unwinding started", "storage":false, "t":"2024-12-17T18:06:53.166870269Z", "to":6.5601807E7}
ERROR 2024-12-17T18:06:53.182577778Z {"codes":false, "from":6.5601832E7, "lvl":"info", "msg":"[8/15 HashState] Unwinding started", "storage":false, "t":"2024-12-17T18:06:53.182319178Z", "to":6.5601807E7}
ERROR 2024-12-17T18:06:53.336297189Z {"expected":"b4226bda6922457130232720b003c1887957acce4d5032e60ccc00431e2e293e", "got":"055fe10231d364206cdc9284080573b5f8e8ef43be9884c209380c6744c0fdbb", "lvl":"warn", "msg":"[bor] End block hash mismatch while whitelisting milestone", "t":"2024-12-17T18:06:53.336055749Z"}
ERROR 2024-12-17T18:06:53.336360339Z {"lvl":"warn", "msg":"[bor] Rewinding chain due to milestone endblock hash mismatch", "number":6.5601807E7, "t":"2024-12-17T18:06:53.336168899Z"}
ERROR 2024-12-17T18:06:53.458250871Z {"codes":false, "from":6.5601832E7, "lvl":"info", "msg":"[8/15 HashState] Unwinding started", "storage":true, "t":"2024-12-17T18:06:53.458036541Z", "to":6.5601807E7}
ERROR 2024-12-17T18:06:54.185370866Z {"csbucket":"AccountChangeSet", "from":6.5601832E7, "lvl":"info", "msg":"[9/15 IntermediateHashes] Unwinding", "t":"2024-12-17T18:06:54.185085206Z", "to":6.5601807E7}
ERROR 2024-12-17T18:06:54.212231084Z {"csbucket":"StorageChangeSet", "from":6.5601832E7, "lvl":"info", "msg":"[9/15 IntermediateHashes] Unwinding", "t":"2024-12-17T18:06:54.211966194Z", "to":6.5601807E7}
ERROR 2024-12-17T18:06:55.369709431Z {"hash":"0x897dfbdc4abb43e59410d3a084e2a3ced4ccfba49156e050653114f53f603c39", "lvl":"info", "msg":"[9/15 IntermediateHashes] Trie root", "t":"2024-12-17T18:06:55.369432081Z"}
ERROR 2024-12-17T18:06:58.175630096Z {"from":6.5601832E7, "lvl":"info", "msg":"[7/15 Execution] Unwind Execution", "t":"2024-12-17T18:06:58.175393106Z", "to":6.5601807E7}
ERROR 2024-12-17T18:06:59.316387305Z {"from":6.5601807E7, "lvl":"info", "msg":"[2/15 Headers] Waiting for headers...", "t":"2024-12-17T18:06:59.316039555Z"}
ERROR 2024-12-17T18:06:59.316481494Z {"age":"55y9mo4d", "blk/sec":9.223372036854776E18, "headers":9.223372036854776E18, "highest":0, "in":2.3029E-4, "lvl":"info", "msg":"[2/15 Headers] Processed", "t":"2024-12-17T18:06:59.316257205Z"}
ERROR 2024-12-17T18:07:00.655101061Z {"lvl":"warn", "msg":"sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback.", "t":"2024-12-17T18:07:00.652658511Z"}
ERROR 2024-12-17T18:07:00.655266371Z {"in":"1.335452386s", "lvl":"info", "msg":"Commit cycle", "t":"2024-12-17T18:07:00.655137741Z"}
ERROR 2024-12-17T18:07:00.655345271Z {"Unwind AccountHistoryIndex":"358ms", "Unwind CallTraces":"1.165s", "Unwind Execution":"1.085s", "Unwind HashState":"1.018s", "Unwind Headers":"55ms", "Unwind IntermediateHashes":"3.99s", "Unwind LogIndex":"1.056s", "Unwind StorageHistoryIndex":"1s", "Unwind TxLookup":"116ms", "lvl":"info", "msg":"Timings (slower than 50ms)", "t":"2024-12-17T18:07:00.655180561Z"}
ERROR 2024-12-17T18:07:00.655369121Z {"AccountChangeSet":"152.1GB", "BlockTransaction":"357.5GB", "FreeList":"46.3MB", "PlainState":"481.4GB", "ReclaimableSpace":"185.3GB", "StorageChangeSet":"463.3GB", "TransactionLog":"2.2TB", "lvl":"info", "msg":"Tables", "t":"2024-12-17T18:07:00.655211691Z"}
ERROR 2024-12-17T18:07:02.402909872Z {"from":6.5601807E7, "lvl":"info", "msg":"[2/15 Headers] Waiting for headers...", "t":"2024-12-17T18:07:02.402635072Z"}
ERROR 2024-12-17T18:07:02.403001122Z {"age":"55y9mo4d", "blk/sec":9.223372036854776E18, "headers":9.223372036854776E18, "highest":0, "in":1.9787E-4, "lvl":"info", "msg":"[2/15 Headers] Processed", "t":"2024-12-17T18:07:02.402820192Z"}
ERROR 2024-12-17T18:07:02.403552782Z {"lvl":"warn", "msg":"sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback.", "t":"2024-12-17T18:07:02.403412562Z"}
ERROR 2024-12-17T18:07:02.403956852Z {"from":6.5601807E7, "lvl":"info", "msg":"[2/15 Headers] Waiting for headers...", "t":"2024-12-17T18:07:02.403848412Z"}
ERROR 2024-12-17T18:07:02.403991242Z {"age":"55y9mo4d", "blk/sec":9.223372036854776E18, "headers":9.223372036854776E18, "highest":0, "in":7.206E-5, "lvl":"info", "msg":"[2/15 Headers] Processed", "t":"2024-12-17T18:07:02.403918792Z"}
ERROR 2024-12-17T18:07:02.404475232Z {"lvl":"warn", "msg":"sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback.", "t":"2024-12-17T18:07:02.404360461Z"}
ERROR 2024-12-17T18:07:02.404752412Z {"from":6.5601807E7, "lvl":"info", "msg":"[2/15 Headers] Waiting for headers...", "t":"2024-12-17T18:07:02.404660652Z"}
ERROR 2024-12-17T18:07:02.404778701Z {"age":"55y9mo4d", "blk/sec":9.223372036854776E18, "headers":9.223372036854776E18, "highest":0, "in":5.4211E-5, "lvl":"info", "msg":"[2/15 Headers] Processed", "t":"2024-12-17T18:07:02.404712681Z"}
ERROR 2024-12-17T18:07:02.405189381Z {"lvl":"warn", "msg":"sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback.", "t":"2024-12-17T18:07:02.405097172Z"}
ERROR 2024-12-17T18:07:02.405502221Z {"from":6.5601807E7, "lvl":"info", "msg":"[2/15 Headers] Waiting for headers...", "t":"2024-12-17T18:07:02.405412052Z"}
ERROR 2024-12-17T18:07:02.405534381Z {"age":"55y9mo4d", "blk/sec":9.223372036854776E18, "headers":9.223372036854776E18, "highest":0, "in":6.6791E-5, "lvl":"info", "msg":"[2/15 Headers] Processed", "t":"2024-12-17T18:07:02.405476172Z"}
ERROR 2024-12-17T18:07:02.405942232Z {"lvl":"warn", "msg":"sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback.", "t":"2024-12-17T18:07:02.405838881Z"}
ERROR 2024-12-17T18:07:02.406201471Z {"from":6.5601807E7, "lvl":"info", "msg":"[2/15 Headers] Waiting for headers...", "t":"2024-12-17T18:07:02.406100721Z"}
ERROR 2024-12-17T18:07:02.406228561Z {"age":"55y9mo4d", "blk/sec":9.223372036854776E18, "headers":9.223372036854776E18, "highest":0, "in":7.026E-5, "lvl":"info", "msg":"[2/15 Headers] Processed", "t":"2024-12-17T18:07:02.406168951Z"}
ERROR 2024-12-17T18:07:02.406709531Z {"lvl":"warn", "msg":"sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback.", "t":"2024-12-17T18:07:02.406602641Z"}
ERROR 2024-12-17T18:07:02.406965101Z {"from":6.5601807E7, "lvl":"info", "msg":"[2/15 Headers] Waiting for headers...", "t":"2024-12-17T18:07:02.406878471Z"}
ERROR 2024-12-17T18:07:02.406994851Z {"age":"55y9mo4d", "blk/sec":9.223372036854776E18, "headers":9.223372036854776E18, "highest":0, "in":6.787E-5, "lvl":"info", "msg":"[2/15 Headers] Processed", "t":"2024-12-17T18:07:02.406942171Z"}

MrFreezeDZ avatar Dec 18 '24 11:12 MrFreezeDZ

I deleted the Erigon pod, which led to a restart of the process. Now it looks to me, as if it works:

{"in":"2h11m6.337873468s", "lvl":"info", "msg":"[7/15 Execution] DONE", "t":"2024-12-18T16:35:30.60788838Z"}

Right now Erigon writes messages like these:

{"current_prefix":"295fdcf4", "into":"HashedStorage", "lvl":"info", "msg":"[8/15 HashState] ETL [2/2] Loading", "t":"2024-12-18T17:11:55.050943029Z"}

MrFreezeDZ avatar Dec 18 '24 17:12 MrFreezeDZ

The issue was resolved by restarting Erigon. I am just curious what the origin of the error was. Until the restart Erigon logged three lines every single millisecond, so I guess here is a hidden problem somewhere. In my opinion this should be looked at, but if the issue gets closed, then so be it.

MrFreezeDZ avatar Dec 20 '24 09:12 MrFreezeDZ

Hello, yesterday Erigon had a very similiar error pattern but without writing logs every millisecond. I put here all log lines I found interesting. For this I filtered out all lines containing following strings:

  • "[2/15 Headers] No block headers to write in this log period"
  • "[mem] memory stats"
  • "[Experiment] heap profile threshold check"
  • "[p2p] GoodPeers"
  • "Req/resp stats"
  • "[txpool] stat"

This results in these Loglines, beginning from the last notification of the RPC Deamon to the reboot, which again seems to resolve the issue:

{"amount":15, "from":6.578162E7, "hash":"0xecd3ffdab70fa3653190c013b0ea87eecdd7cba7ffb88e492688ba0052879587", "header sending":"64.41µs", "log sending":"420ns", "lvl":"info", "msg":"RPC Daemon notified of new headers", "t":"2024-12-22T06:10:44.774227876Z", "to":6.5781635E7}
{"in":"1.14334632s", "lvl":"info", "msg":"Commit cycle", "t":"2024-12-22T06:10:44.774431646Z"}
{"AccountHistoryIndex":"383ms", "Bodies":"54ms", "BorHeimdall":"99ms", "CallTraces":"2.152s", "Execution":"5.632s", "HashState":"4.808s", "Headers":"109ms", "IntermediateHashes":"9.701s", "LogIndex":"3.391s", "StorageHistoryIndex":"3.255s", "TxLookup":"1.573s", "Unwind LogIndex":"74ms", "Unwind StorageHistoryIndex":"84ms", "lvl":"info", "msg":"Timings (slower than 50ms)", "t":"2024-12-22T06:10:44.774466246Z"}
{"AccountChangeSet":"153.6GB", "BlockTransaction":"369.6GB", "FreeList":"70.2MB", "PlainState":"482.9GB", "ReclaimableSpace":"280.9GB", "StorageChangeSet":"468.4GB", "TransactionLog":"2.3TB", "lvl":"info", "msg":"Tables", "t":"2024-12-22T06:10:44.774522176Z"}
{"from":6.5781635E7, "lvl":"info", "msg":"[2/15 Headers] Waiting for headers...", "t":"2024-12-22T06:10:45.35716514Z"}
{"age":"55y9mo1w", "blk/sec":9.223372036854776E18, "headers":9.223372036854776E18, "highest":0, "in":2.609E-4, "lvl":"info", "msg":"[2/15 Headers] Processed", "t":"2024-12-22T06:10:45.35740465Z"}
{"err":"[3/15 BorHeimdall] verification failed for header 65781635: ecd3ffdab70fa3653190c013b0ea87eecdd7cba7ffb88e492688ba0052879587", "lvl":"eror", "msg":"Staged Sync", "t":"2024-12-22T06:10:45.35761984Z"}
{"from":6.5781635E7, "lvl":"info", "msg":"[10/15 CallTraces] Unwind", "t":"2024-12-22T06:10:54.178470365Z", "to":6.5781609E7}
{"codes":true, "from":6.5781635E7, "lvl":"info", "msg":"[8/15 HashState] Unwinding started", "storage":false, "t":"2024-12-22T06:10:56.063692018Z", "to":6.5781609E7}
{"codes":false, "from":6.5781635E7, "lvl":"info", "msg":"[8/15 HashState] Unwinding started", "storage":false, "t":"2024-12-22T06:10:56.074260937Z", "to":6.5781609E7}
{"codes":false, "from":6.5781635E7, "lvl":"info", "msg":"[8/15 HashState] Unwinding started", "storage":true, "t":"2024-12-22T06:10:56.231117768Z", "to":6.5781609E7}
{"csbucket":"AccountChangeSet", "from":6.5781635E7, "lvl":"info", "msg":"[9/15 IntermediateHashes] Unwinding", "t":"2024-12-22T06:10:57.013715819Z", "to":6.5781609E7}
{"csbucket":"StorageChangeSet", "from":6.5781635E7, "lvl":"info", "msg":"[9/15 IntermediateHashes] Unwinding", "t":"2024-12-22T06:10:57.027778208Z", "to":6.5781609E7}
{"hash":"0x994ece3878572b9b1c9106c660e13406e5abf29492a0b9abc866df10ff295a9c", "lvl":"info", "msg":"[9/15 IntermediateHashes] Trie root", "t":"2024-12-22T06:10:57.675579119Z"}
{"from":6.5781635E7, "lvl":"info", "msg":"[7/15 Execution] Unwind Execution", "t":"2024-12-22T06:10:59.76407828Z", "to":6.5781609E7}
{"from":6.5781609E7, "lvl":"info", "msg":"[2/15 Headers] Waiting for headers...", "t":"2024-12-22T06:11:00.583858029Z"}
{"end":6.5780133E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0x511a4d4f7cc535cefd131cf0f9814e3e2cf13542985442e0dd681f8d3e742660", "start":6.5779622E7, "t":"2024-12-22T06:11:22.353583259Z"}
{"end":6.5780133E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0x511a4d4f7cc535cefd131cf0f9814e3e2cf13542985442e0dd681f8d3e742660", "start":6.5779622E7, "t":"2024-12-22T06:13:03.135350176Z"}
{"end":6.5780133E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0x511a4d4f7cc535cefd131cf0f9814e3e2cf13542985442e0dd681f8d3e742660", "start":6.5779622E7, "t":"2024-12-22T06:14:42.350937454Z"}
{"end":6.5780133E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0x511a4d4f7cc535cefd131cf0f9814e3e2cf13542985442e0dd681f8d3e742660", "start":6.5779622E7, "t":"2024-12-22T06:16:22.353616472Z"}
{"end":6.5780645E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0x0259bdd6868f0542aa860cf74cd8802cca47838d2ad7a78d535d8bbddcf972e9", "start":6.5780134E7, "t":"2024-12-22T06:18:02.356585069Z"}
{"end":6.5780645E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0x0259bdd6868f0542aa860cf74cd8802cca47838d2ad7a78d535d8bbddcf972e9", "start":6.5780134E7, "t":"2024-12-22T06:19:42.353056381Z"}
{"end":6.5780645E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0x0259bdd6868f0542aa860cf74cd8802cca47838d2ad7a78d535d8bbddcf972e9", "start":6.5780134E7, "t":"2024-12-22T06:21:22.353715097Z"}
{"end":6.5780645E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0x0259bdd6868f0542aa860cf74cd8802cca47838d2ad7a78d535d8bbddcf972e9", "start":6.5780134E7, "t":"2024-12-22T06:23:02.353670624Z"}
{"end":6.5780645E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0x0259bdd6868f0542aa860cf74cd8802cca47838d2ad7a78d535d8bbddcf972e9", "start":6.5780134E7, "t":"2024-12-22T06:24:42.351470174Z"}
{"end":6.5780645E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0x0259bdd6868f0542aa860cf74cd8802cca47838d2ad7a78d535d8bbddcf972e9", "start":6.5780134E7, "t":"2024-12-22T06:26:22.353860948Z"}
{"end":6.5780645E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0x0259bdd6868f0542aa860cf74cd8802cca47838d2ad7a78d535d8bbddcf972e9", "start":6.5780134E7, "t":"2024-12-22T06:28:02.354899501Z"}
{"end":6.5780645E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0x0259bdd6868f0542aa860cf74cd8802cca47838d2ad7a78d535d8bbddcf972e9", "start":6.5780134E7, "t":"2024-12-22T06:29:42.353129463Z"}
{"end":6.5780645E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0x0259bdd6868f0542aa860cf74cd8802cca47838d2ad7a78d535d8bbddcf972e9", "start":6.5780134E7, "t":"2024-12-22T06:31:22.353825152Z"}
{"end":6.5780645E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0x0259bdd6868f0542aa860cf74cd8802cca47838d2ad7a78d535d8bbddcf972e9", "start":6.5780134E7, "t":"2024-12-22T06:33:02.353092833Z"}
{"end":6.5780645E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0x0259bdd6868f0542aa860cf74cd8802cca47838d2ad7a78d535d8bbddcf972e9", "start":6.5780134E7, "t":"2024-12-22T06:34:42.351339451Z"}
{"end":6.5781157E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0xb502124424f67ccb050805128af2773319ea3272e527e223da6f45fd5c7d1da4", "start":6.5780646E7, "t":"2024-12-22T06:36:22.356654822Z"}
{"end":6.5781157E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0xb502124424f67ccb050805128af2773319ea3272e527e223da6f45fd5c7d1da4", "start":6.5780646E7, "t":"2024-12-22T06:38:02.354220159Z"}
{"end":6.5781157E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0xb502124424f67ccb050805128af2773319ea3272e527e223da6f45fd5c7d1da4", "start":6.5780646E7, "t":"2024-12-22T06:39:42.353854429Z"}
{"end":6.5781157E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0xb502124424f67ccb050805128af2773319ea3272e527e223da6f45fd5c7d1da4", "start":6.5780646E7, "t":"2024-12-22T06:41:22.354082812Z"}
{"end":6.5781157E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0xb502124424f67ccb050805128af2773319ea3272e527e223da6f45fd5c7d1da4", "start":6.5780646E7, "t":"2024-12-22T06:43:03.142126055Z"}
{"end":6.5781157E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0xb502124424f67ccb050805128af2773319ea3272e527e223da6f45fd5c7d1da4", "start":6.5780646E7, "t":"2024-12-22T06:44:42.351017255Z"}
{"end":6.5781157E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0xb502124424f67ccb050805128af2773319ea3272e527e223da6f45fd5c7d1da4", "start":6.5780646E7, "t":"2024-12-22T06:46:22.352693542Z"}
{"end":6.5781157E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0xb502124424f67ccb050805128af2773319ea3272e527e223da6f45fd5c7d1da4", "start":6.5780646E7, "t":"2024-12-22T06:48:02.353243246Z"}
{"end":6.5781157E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0xb502124424f67ccb050805128af2773319ea3272e527e223da6f45fd5c7d1da4", "start":6.5780646E7, "t":"2024-12-22T06:49:42.353739023Z"}
{"end":6.5781157E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0xb502124424f67ccb050805128af2773319ea3272e527e223da6f45fd5c7d1da4", "start":6.5780646E7, "t":"2024-12-22T06:51:22.352817894Z"}
{"end":6.5781157E7, "lvl":"info", "msg":"[bor.heimdall] Got new checkpoint", "rootHash":"0xb502124424f67ccb050805128af2773319ea3272e527e223da6f45fd5c7d1da4", "start":6.5780646E7, "t":"2024-12-22T06:53:02.35259081Z"}
{"lvl":"info", "msg":"Got interrupt, shutting down...", "sig":"terminated", "t":"2024-12-23T09:26:35.972418203Z"}
{"lvl":"info", "msg":"Got interrupt, shutting down...", "t":"2024-12-23T09:26:35.972421943Z"}
{"lvl":"info", "msg":"Exiting...", "t":"2024-12-23T09:26:35.972986712Z"}
{"lvl":"info", "msg":"HTTP endpoint closed", "t":"2024-12-23T09:26:35.973187263Z", "url":"[::]:8545"}
{"lvl":"info", "msg":"HTTP endpoint closed", "t":"2024-12-23T09:26:35.973327932Z", "url":"[::]:8546"}
{"lvl":"info", "msg":"RPC server shutting down", "t":"2024-12-23T09:26:35.973400143Z"}
[dbg] env SAVE_HEAP_PROFILE=true
{"file prefix":"erigon", "json":false, "log dir":"/data/logs", "log level":"info", "lvl":"info", "msg":"logging to file system", "t":"2024-12-23T09:29:07.768520814Z"}
{"lvl":"info", "msg":"Enabling metrics export to prometheus", "path":"http://0.0.0.0:6060/debug/metrics/prometheus", "t":"2024-12-23T09:29:07.777446893Z"}
{"git_branch":"heads/2.60.8", "git_commit":"dec9d2067be63cb155016efa05050339e2a8994a", "git_tag":"v2.60.6-32-gdec9d2067b-dirty", "lvl":"info", "msg":"Build info", "t":"2024-12-23T09:29:07.777501793Z"}
{"lvl":"info", "msg":"Starting Erigon on Bor Mainnet...", "t":"2024-12-23T09:29:07.777538723Z"}
{"ETH":500, "lvl":"info", "msg":"Maximum peer count", "t":"2024-12-23T09:29:07.779305833Z", "total":500}
{"APIs":"eth,debug,trace", "lvl":"info", "msg":"starting HTTP APIs", "port":8545, "t":"2024-12-23T09:29:07.779483863Z"}
{"level":"WRN", "lvl":"info", "msg":"torrent verbosity", "t":"2024-12-23T09:29:07.779713173Z"}
{"cap":5.0E7, "lvl":"info", "msg":"Set global gas cap", "t":"2024-12-23T09:29:09.884691023Z"}
{"download.rate":"300mb", "ipv4-enabled":true, "ipv6-enabled":true, "lvl":"info", "msg":"[Downloader] Running with", "t":"2024-12-23T09:29:09.886476623Z", "upload.rate":"4mb"}
{"label":"chaindata", "lvl":"info", "msg":"Opening Database", "path":"/data/chaindata", "t":"2024-12-23T09:29:09.886653953Z"}
{"label":"chaindata", "lvl":"info", "msg":"[db] open", "pageSize":16384, "sizeLimit":"7TB", "t":"2024-12-23T09:29:09.917531551Z"}
{"lvl":"warn", "msg":"Sanitizing invalid bor miner gas price", "provided":"1000000000", "t":"2024-12-23T09:29:10.044569473Z", "updated":"25000000000"}
{"lvl":"warn", "msg":"Sanitizing invalid bor min fee cap", "provided":1, "t":"2024-12-23T09:29:10.044647073Z", "updated":2.5E10}
{"config":"{ChainID: 137, Homestead: 0, DAO: <nil>, Tangerine Whistle: 0, Spurious Dragon: 0, Byzantium: 0, Constantinople: 0, Petersburg: 0, Istanbul: 3395000, Muir Glacier: 3395000, Berlin: 14750000, London: 23850000, Arrow Glacier: <nil>, Gray Glacier: <nil>, Terminal Total Difficulty: <nil>, Merge Netsplit: <nil>, Shanghai: <nil>, Cancun: <nil>, Prague: <nil>, Osaka: <nil>, Engine: bor, NoPruneContracts: map[]}", "genesis":"0xa9c28ce2141b56c474f1dc504bee9b01eb1bd7d1a507580d5519d4437a97de1b", "lvl":"info", "msg":"Initialised chain configuration", "t":"2024-12-23T09:29:10.044709423Z"}
{"lvl":"info", "msg":"Initialising Ethereum protocol", "network":137, "t":"2024-12-23T09:30:12.145888954Z"}
{"label":"bor", "lvl":"info", "msg":"Opening Database", "path":"/data/bor", "t":"2024-12-23T09:30:12.146073524Z"}
{"lvl":"info", "msg":"Starting private RPC server", "on":"127.0.0.1:9090", "t":"2024-12-23T09:30:22.623538976Z"}
{"lvl":"info", "msg":"new subscription to logs established", "t":"2024-12-23T09:30:22.623636736Z"}
{"lvl":"info", "msg":"rpc filters: subscribing to Erigon events", "t":"2024-12-23T09:30:22.623924096Z"}
{"lvl":"info", "msg":"new subscription to newHeaders established", "t":"2024-12-23T09:30:22.624248396Z"}
{"lvl":"info", "msg":"New txs subscriber joined", "t":"2024-12-23T09:30:22.624249576Z"}
{"grpc":false, "http.url":"[::]:8545", "lvl":"info", "msg":"JsonRpc endpoint opened", "t":"2024-12-23T09:30:22.625008056Z", "websocket.url":"[::]:8546", "ws":true, "ws.compression":true}
{"lvl":"info", "msg":"[txpool] Started", "t":"2024-12-23T09:30:22.641974415Z"}
{"lvl":"info", "msg":"[1/15 Snapshots] Requesting downloads", "t":"2024-12-23T09:30:22.683709272Z"}
{"err":"no UPnP or NAT-PMP router discovered", "lvl":"warn", "msg":"NAT ExternalIP resolution has failed, try to pass a different --nat option", "t":"2024-12-23T09:30:22.697047181Z"}
{"lvl":"info", "msg":"Started P2P networking", "name":"erigon/v2.60.8-dec9d206/linux-amd64/go1.21.5", "self":"enode://6b7469e82e1699591029807cdce3cd372098340f122de38c912cfeb8ff2a35839cb22d99f6593998fadf4fa733e4c12b104367894506bcf55bf9bb8cf815c469@127.0.0.1:30303", "t":"2024-12-23T09:30:22.698605811Z", "version":68}
{"err":"no UPnP or NAT-PMP router discovered", "lvl":"warn", "msg":"NAT ExternalIP resolution has failed, try to pass a different --nat option", "t":"2024-12-23T09:30:22.700443241Z"}
{"lvl":"info", "msg":"Started P2P networking", "name":"erigon/v2.60.8-dec9d206/linux-amd64/go1.21.5", "self":"enode://6b7469e82e1699591029807cdce3cd372098340f122de38c912cfeb8ff2a35839cb22d99f6593998fadf4fa733e4c12b104367894506bcf55bf9bb8cf815c469@127.0.0.1:30304", "t":"2024-12-23T09:30:22.702762211Z", "version":67}
{"lvl":"warn", "max_in_snapshots":5.9699999E7, "min_in_db":5.987E7, "msg":"[1/15 Snapshots] Some blocks are not in snapshots and not in db", "t":"2024-12-23T09:30:27.310978996Z"}
{"alloc":"4.2GB", "blocks":"59700k", "indices":"59700k", "lvl":"info", "msg":"[snapshots:download] Stat", "sys":"7.7GB", "t":"2024-12-23T09:30:27.311300236Z"}
{"from":6.5781635E7, "lvl":"info", "msg":"[2/15 Headers] Waiting for headers...", "t":"2024-12-23T09:30:27.311488556Z"}
{"lvl":"info", "msg":"[2/15 Headers] Inserting headers", "progress":6.5793596E7, "queue":1, "t":"2024-12-23T09:30:57.311636852Z"}
{"alloc":"4.7GB", "blk/second":900.3333333333334, "invalidHeaders":0, "lvl":"info", "msg":"[2/15 Headers] Wrote block headers", "number":6.5808645E7, "rejectedBadHeaders":0, "sys":"7.7GB", "t":"2024-12-23T09:31:27.312548924Z"}
{"alloc":"5.2GB", "blk/second":580.0333333333333, "invalidHeaders":0, "lvl":"info", "msg":"[2/15 Headers] Wrote block headers", "number":6.5826046E7, "rejectedBadHeaders":0, "sys":"7.7GB", "t":"2024-12-23T09:31:57.312447572Z"}

MrFreezeDZ avatar Dec 23 '24 09:12 MrFreezeDZ

Hello, on 2024-12-29T04:11:10.705482522Z we experienced the behavior again. This time again Erigon did not write logs every millisecond. After the manual restart Erigon works again without problems, so I did not add log lines until now. I only attach here the log lines before and after the restart.

Here are the filtered log lines, where I again filtered out all lines containing strings like these:

  • "[2/15 Headers] No block headers to write in this log period"
  • "[mem] memory stats"
  • "[Experiment] heap profile threshold check"
  • "[p2p] GoodPeers"
  • "Req/resp stats"
  • "[txpool] stat"
{"msg":"[bor] End block hash mismatch while whitelisting milestone","lvl":"warn","got":"cb76b68e0af49f4c92cdcbe9bdb458b1133794eb5a55f6759add1a577adb9f6c","t":"2024-12-29T04:11:10.705482522Z","expected":"e5e8fd48773b2ea2b2f808514658e7f60e2586308cd4a9a64e2e90ed08cd84ef"}
{"lvl":"warn","msg":"[bor] Rewinding chain due to milestone endblock hash mismatch","number":66049661,"t":"2024-12-29T04:11:10.705579412Z"}
{"t":"2024-12-29T04:11:10.723064151Z","msg":"RPC Daemon notified of new headers","amount":23,"lvl":"info","header sending":"66.48µs","log sending":"120ns","from":66049679,"hash":"0xc5598d34c4a09c66037081886909acaf2f6220020367f668359a9545f788d92e","to":66049702}
{"msg":"Commit cycle","in":"1.031769407s","lvl":"info","t":"2024-12-29T04:11:10.723216671Z"}
{"CallTraces":"1.02s","Unwind Execution":"713ms","TxLookup":"3.136s","IntermediateHashes":"6.69s","BorHeimdall":"132ms","StorageHistoryIndex":"2.067s","Execution":"4.103s","AccountHistoryIndex":"417ms","Unwind LogIndex":"1.073s","t":"2024-12-29T04:11:10.723256951Z","msg":"Timings (slower than 50ms)","Unwind IntermediateHashes":"1.901s","Unwind HashState":"764ms","lvl":"info","Unwind CallTraces":"1.05s","Unwind AccountHistoryIndex":"283ms","Unwind StorageHistoryIndex":"743ms","Headers":"217ms","LogIndex":"2.796s","HashState":"3.614s","Bodies":"61ms"}
{"msg":"Tables","AccountChangeSet":"155.5GB","StorageChangeSet":"475.6GB","BlockTransaction":"388.4GB","FreeList":"62.3MB","PlainState":"484.9GB","ReclaimableSpace":"249.2GB","lvl":"info","TransactionLog":"2.3TB","t":"2024-12-29T04:11:10.723336261Z"}
{"t":"2024-12-29T04:11:11.06685199Z","from":66049702,"msg":"[2/15 Headers] Waiting for headers...","lvl":"info"}
{"t":"2024-12-29T04:11:11.06703705Z","msg":"[2/15 Headers] Processed","lvl":"info","highest":0,"age":"55y9mo2w","in":0.00020149,"headers":9223372036854776000,"blk/sec":9223372036854776000}
{"err":"[3/15 BorHeimdall] verification failed for header 66049702: c5598d34c4a09c66037081886909acaf2f6220020367f668359a9545f788d92e","msg":"Staged Sync","t":"2024-12-29T04:11:11.06802415Z","lvl":"eror"}
{"lvl":"info","to":66049661,"from":66049702,"msg":"[10/15 CallTraces] Unwind","t":"2024-12-29T04:11:20.597342945Z"}
{"t":"2024-12-29T04:11:22.127378331Z","from":66049702,"lvl":"info","codes":true,"storage":false,"msg":"[8/15 HashState] Unwinding started","to":66049661}
{"lvl":"info","codes":false,"storage":false,"msg":"[8/15 HashState] Unwinding started","t":"2024-12-29T04:11:22.14079056Z","from":66049702,"to":66049661}
{"storage":true,"lvl":"info","from":66049702,"msg":"[8/15 HashState] Unwinding started","t":"2024-12-29T04:11:22.389888635Z","to":66049661,"codes":false}
{"t":"2024-12-29T04:11:23.899985672Z","lvl":"info","csbucket":"AccountChangeSet","from":66049702,"to":66049661,"msg":"[9/15 IntermediateHashes] Unwinding"}
{"csbucket":"StorageChangeSet","lvl":"info","msg":"[9/15 IntermediateHashes] Unwinding","from":66049702,"to":66049661,"t":"2024-12-29T04:11:23.92784617Z"}
{"hash":"0xfb382cde5d68b14d9ad5cd6efb6ffcc888b4d767928cc6ffe9dec242dce9a48b","msg":"[9/15 IntermediateHashes] Trie root","lvl":"info","t":"2024-12-29T04:11:25.414207789Z"}
{"from":66049702,"msg":"[7/15 Execution] Unwind Execution","t":"2024-12-29T04:11:28.292329672Z","to":66049661,"lvl":"info"}
{"lvl":"info","msg":"[2/15 Headers] Waiting for headers...","from":66049661,"t":"2024-12-29T04:11:29.687866736Z"}
{"sig":"terminated","msg":"Got interrupt, shutting down...","lvl":"info","t":"2025-01-02T08:52:33.50186888Z"}
{"t":"2025-01-02T08:52:33.50189463Z","msg":"Got interrupt, shutting down...","lvl":"info"}
{"lvl":"info","t":"2025-01-02T08:52:33.50243201Z","msg":"Exiting..."}
{"t":"2025-01-02T08:52:33.50277009Z","url":"[::]:8545","msg":"HTTP endpoint closed","lvl":"info"}
{"t":"2025-01-02T08:52:33.50291348Z","msg":"HTTP endpoint closed","lvl":"info","url":"[::]:8546"}
{"lvl":"info","t":"2025-01-02T08:52:33.50294863Z","msg":"RPC server shutting down"}
[dbg] env SAVE_HEAP_PROFILE=true
{"t":"2025-01-02T08:55:08.892823532Z","log level":"info","lvl":"info","json":false,"msg":"logging to file system","file prefix":"erigon","log dir":"/data/logs"}
{"t":"2025-01-02T08:55:08.906687281Z","path":"http://0.0.0.0:6060/debug/metrics/prometheus","msg":"Enabling metrics export to prometheus","lvl":"info"}
{"git_tag":"v2.60.6-32-gdec9d2067b-dirty","lvl":"info","msg":"Build info","git_branch":"heads/2.60.8","git_commit":"dec9d2067be63cb155016efa05050339e2a8994a","t":"2025-01-02T08:55:08.906765071Z"}
{"t":"2025-01-02T08:55:08.906799251Z","msg":"Starting Erigon on Bor Mainnet...","lvl":"info"}
{"ETH":500,"lvl":"info","total":500,"msg":"Maximum peer count","t":"2025-01-02T08:55:08.908688101Z"}
{"t":"2025-01-02T08:55:08.908867551Z","APIs":"eth,debug,trace","msg":"starting HTTP APIs","port":8545,"lvl":"info"}
{"msg":"torrent verbosity","lvl":"info","level":"WRN","t":"2025-01-02T08:55:08.909090961Z"}
{"cap":50000000,"t":"2025-01-02T08:55:11.014343481Z","msg":"Set global gas cap","lvl":"info"}
{"msg":"[Downloader] Running with","upload.rate":"4mb","ipv4-enabled":true,"t":"2025-01-02T08:55:11.016180521Z","lvl":"info","ipv6-enabled":true,"download.rate":"300mb"}
{"lvl":"info","msg":"Opening Database","t":"2025-01-02T08:55:11.016419701Z","label":"chaindata","path":"/data/chaindata"}
{"label":"chaindata","pageSize":16384,"t":"2025-01-02T08:55:11.04805108Z","msg":"[db] open","sizeLimit":"7TB","lvl":"info"}
{"lvl":"warn","t":"2025-01-02T08:55:11.221014559Z","provided":"1000000000","msg":"Sanitizing invalid bor miner gas price","updated":"25000000000"}
{"msg":"Sanitizing invalid bor min fee cap","updated":25000000000,"provided":1,"lvl":"warn","t":"2025-01-02T08:55:11.221115249Z"}
{"lvl":"info","genesis":"0xa9c28ce2141b56c474f1dc504bee9b01eb1bd7d1a507580d5519d4437a97de1b","config":"{ChainID: 137, Homestead: 0, DAO: <nil>, Tangerine Whistle: 0, Spurious Dragon: 0, Byzantium: 0, Constantinople: 0, Petersburg: 0, Istanbul: 3395000, Muir Glacier: 3395000, Berlin: 14750000, London: 23850000, Arrow Glacier: <nil>, Gray Glacier: <nil>, Terminal Total Difficulty: <nil>, Merge Netsplit: <nil>, Shanghai: <nil>, Cancun: <nil>, Prague: <nil>, Osaka: <nil>, Engine: bor, NoPruneContracts: map[]}","msg":"Initialised chain configuration","t":"2025-01-02T08:55:11.221201799Z"}
{"lvl":"info","t":"2025-01-02T08:56:13.349605579Z","network":137,"msg":"Initialising Ethereum protocol"}
{"label":"bor","t":"2025-01-02T08:56:13.349780779Z","msg":"Opening Database","lvl":"info","path":"/data/bor"}
{"on":"127.0.0.1:9090","lvl":"info","msg":"Starting private RPC server","t":"2025-01-02T08:56:24.803727031Z"}
{"lvl":"info","t":"2025-01-02T08:56:24.803831011Z","msg":"new subscription to logs established"}
{"msg":"rpc filters: subscribing to Erigon events","lvl":"info","t":"2025-01-02T08:56:24.804377251Z"}
{"t":"2025-01-02T08:56:24.804722941Z","lvl":"info","msg":"new subscription to newHeaders established"}
{"msg":"New txs subscriber joined","lvl":"info","t":"2025-01-02T08:56:24.804854811Z"}
{"msg":"JsonRpc endpoint opened","lvl":"info","ws":true,"ws.compression":true,"websocket.url":"[::]:8546","grpc":false,"t":"2025-01-02T08:56:24.806069511Z","http.url":"[::]:8545"}
{"lvl":"info","msg":"[txpool] Started","t":"2025-01-02T08:56:24.837477529Z"}
{"lvl":"info","msg":"[1/15 Snapshots] Requesting downloads","t":"2025-01-02T08:56:24.873137087Z"}
{"lvl":"warn","msg":"NAT ExternalIP resolution has failed, try to pass a different --nat option","err":"no UPnP or NAT-PMP router discovered","t":"2025-01-02T08:56:24.887888536Z"}
{"lvl":"warn","t":"2025-01-02T08:56:24.889102976Z","err":"no UPnP or NAT-PMP router discovered","msg":"NAT ExternalIP resolution has failed, try to pass a different --nat option"}
{"t":"2025-01-02T08:56:24.889509246Z","self":"enode://6b7469e82e1699591029807cdce3cd372098340f122de38c912cfeb8ff2a35839cb22d99f6593998fadf4fa733e4c12b104367894506bcf55bf9bb8cf815c469@127.0.0.1:30303","msg":"Started P2P networking","name":"erigon/v2.60.8-dec9d206/linux-amd64/go1.21.5","version":68,"lvl":"info"}
{"lvl":"info","msg":"Started P2P networking","version":67,"name":"erigon/v2.60.8-dec9d206/linux-amd64/go1.21.5","t":"2025-01-02T08:56:24.890922426Z","self":"enode://6b7469e82e1699591029807cdce3cd372098340f122de38c912cfeb8ff2a35839cb22d99f6593998fadf4fa733e4c12b104367894506bcf55bf9bb8cf815c469@127.0.0.1:30304"}
{"lvl":"warn","t":"2025-01-02T08:56:29.382026349Z","min_in_db":59870000,"max_in_snapshots":59699999,"msg":"[1/15 Snapshots] Some blocks are not in snapshots and not in db"}
{"alloc":"4.2GB","blocks":"59700k","lvl":"info","t":"2025-01-02T08:56:29.382516109Z","msg":"[snapshots:download] Stat","sys":"7.7GB","indices":"59700k"}
{"lvl":"info","msg":"[2/15 Headers] Waiting for headers...","from":66049702,"t":"2025-01-02T08:56:29.382691009Z"}
{"invalidHeaders":0,"rejectedBadHeaders":0,"number":66061278,"t":"2025-01-02T08:56:59.384221103Z","alloc":"4.7GB","lvl":"info","msg":"[2/15 Headers] Wrote block headers","blk/second":385.8666666666667,"sys":"7.7GB"}
{"t":"2025-01-02T08:57:29.383895564Z","msg":"[2/15 Headers] Wrote block headers","blk/second":482.3333333333333,"alloc":"5.2GB","invalidHeaders":0,"rejectedBadHeaders":0,"sys":"7.7GB","lvl":"info","number":66075748}
{"t":"2025-01-02T08:57:59.384452353Z","invalidHeaders":0,"msg":"[2/15 Headers] Wrote block headers","sys":"7.7GB","rejectedBadHeaders":0,"alloc":"5.6GB","lvl":"info","number":66090025,"blk/second":475.9}
{"alloc":"5.6GB","rejectedBadHeaders":0,"t":"2025-01-02T08:58:29.383921191Z","number":66104495,"invalidHeaders":0,"lvl":"info","sys":"7.7GB","msg":"[2/15 Headers] Wrote block headers","blk/second":482.3333333333333}
{"blk/second":482.3333333333333,"sys":"7.7GB","msg":"[2/15 Headers] Wrote block headers","t":"2025-01-02T08:58:59.384018059Z","number":66118965,"invalidHeaders":0,"lvl":"info","rejectedBadHeaders":0,"alloc":"4.2GB"}
{"invalidHeaders":0,"alloc":"4.7GB","t":"2025-01-02T08:59:29.383737606Z","rejectedBadHeaders":0,"lvl":"info","blk/second":475.9,"sys":"7.7GB","msg":"[2/15 Headers] Wrote block headers","number":66133242}
{"alloc":"5.1GB","msg":"[2/15 Headers] Wrote block headers","number":66147712,"rejectedBadHeaders":0,"lvl":"info","sys":"7.7GB","t":"2025-01-02T08:59:59.383808493Z","invalidHeaders":0,"blk/second":482.3333333333333}
{"number":66161603,"t":"2025-01-02T09:00:29.384415308Z","alloc":"5.6GB","lvl":"info","sys":"7.7GB","invalidHeaders":0,"rejectedBadHeaders":0,"blk/second":463.03333333333336,"msg":"[2/15 Headers] Wrote block headers"}
{"sys":"7.7GB","blk/second":463.03333333333336,"alloc":"4.2GB","invalidHeaders":0,"msg":"[2/15 Headers] Wrote block headers","t":"2025-01-02T09:00:59.38413553Z","lvl":"info","number":66175494,"rejectedBadHeaders":0}
{"blk/second":469.46666666666664,"alloc":"4.6GB","sys":"7.7GB","lvl":"info","t":"2025-01-02T09:01:29.384640281Z","invalidHeaders":0,"msg":"[2/15 Headers] Wrote block headers","rejectedBadHeaders":0,"number":66189578}
{"number":66212009,"invalidHeaders":0,"t":"2025-01-02T09:01:59.384341523Z","msg":"[2/15 Headers] Wrote block headers","rejectedBadHeaders":0,"lvl":"info","blk/second":747.7,"alloc":"5.1GB","sys":"7.7GB"}
{"blk/sec":369,"in":439.208190049,"msg":"[2/15 Headers] Processed","t":"2025-01-02T09:03:48.590894068Z","headers":162499,"lvl":"info","highest":66212201,"age":"2s"}
{"msg":"[2/15 Headers] DONE","t":"2025-01-02T09:03:48.591005298Z","lvl":"info","in":"7m19.208350109s"}
{"from":10322,"msg":"[3/15 BorHeimdall] Processing spans...","to":10346,"lvl":"info","t":"2025-01-02T09:03:48.598651448Z"}
{"from":66049703,"t":"2025-01-02T09:03:48.786451796Z","msg":"[3/15 BorHeimdall] Processing sync events...","lvl":"info","to":66212201}
{"lastCheckpointId":0,"t":"2025-01-02T09:04:18.786686711Z","total records":947,"waypoints":"0s","sync-events":"11.239623653s","snap-init":"2.001644027s","lvl":"info","msg":"[3/15 BorHeimdall] StateSync Progress","sync-event-fetch":"10.11967601s","process time":"30.195607383s","snaps":"34.239976603s","lastMilestoneId":0,"progress":66159256,"lastSpanID":10346,"lastStateSyncEventID":3002632}
{"fetch time":"14.008406054s","lvl":"info","lastMilestoneId":0,"sync event time":"15.853856609s","msg":"[3/15 BorHeimdall] Sync events processed","total records":1284,"waypoint time":"0s","lastSpanID":10346,"lastStateSyncEventID":3002969,"snap time":"50.620208148s","t":"2025-01-02T09:04:32.375813021Z","process time":"43.784733923s","lastCheckpointId":0,"progress":66212201}
{"into":"HeaderNumber","current_prefix":"1d9cb211","msg":"[4/15 BlockHashes] ETL [2/2] Loading","t":"2025-01-02T09:05:02.596002003Z","lvl":"info"}
{"lvl":"info","msg":"[4/15 BlockHashes] ETL [2/2] Loading","into":"HeaderNumber","current_prefix":"44153c36","t":"2025-01-02T09:05:32.59159921Z"}
{"msg":"[4/15 BlockHashes] ETL [2/2] Loading","current_prefix":"6d11380e","t":"2025-01-02T09:06:02.590402908Z","into":"HeaderNumber","lvl":"info"}
{"msg":"[4/15 BlockHashes] ETL [2/2] Loading","lvl":"info","current_prefix":"9973ad63","t":"2025-01-02T09:06:32.593992387Z","into":"HeaderNumber"}
{"lvl":"info","into":"HeaderNumber","current_prefix":"c399aca3","msg":"[4/15 BlockHashes] ETL [2/2] Loading","t":"2025-01-02T09:07:02.590894247Z"}
{"t":"2025-01-02T09:07:32.589867317Z","current_prefix":"ed0b018e","msg":"[4/15 BlockHashes] ETL [2/2] Loading","into":"HeaderNumber","lvl":"info"}
{"lvl":"info","msg":"[4/15 BlockHashes] DONE","in":"3m13.188343727s","t":"2025-01-02T09:07:45.564319776Z"}
{"msg":"[5/15 Bodies] Processing bodies...","from":66049702,"lvl":"info","t":"2025-01-02T09:07:45.564566187Z","to":66212201}
{"blk/sec":64,"lvl":"info","block_num":66051141,"delivery/sec":"3.8MB","t":"2025-01-02T09:08:15.565208668Z","sys":"7.9GB","alloc":"4.8GB","remaining":161059,"wasted/sec":"2.4MB","cache":"31.3MB","msg":"[5/15 Bodies] Downloading block bodies","delivered":1929}
{"remaining":160381,"block_num":66051819,"msg":"[5/15 Bodies] Downloading block bodies","lvl":"info","t":"2025-01-02T09:08:45.566311078Z","blk/sec":101,"delivery/sec":"2.3MB","wasted/sec":"3.8MB","delivered":3053,"alloc":"4.2GB","sys":"8.3GB","cache":"54.4MB"}
{"block_num":66053470,"t":"2025-01-02T09:09:15.566044018Z","delivered":4007,"cache":"14.9MB","lvl":"info","alloc":"5.1GB","blk/sec":133,"delivery/sec":"1.8MB","remaining":158730,"msg":"[5/15 Bodies] Downloading block bodies","wasted/sec":"4.3MB","sys":"8.3GB"}
{"lvl":"info","remaining":157846,"sys":"8.6GB","delivery/sec":"1.8MB","wasted/sec":"4.1MB","block_num":66054354,"msg":"[5/15 Bodies] Downloading block bodies","delivered":4978,"blk/sec":165,"cache":"19.3MB","t":"2025-01-02T09:09:45.576796176Z","alloc":"6.0GB"}
{"delivery/sec":"1.7MB","lvl":"info","msg":"[5/15 Bodies] Downloading block bodies","blk/sec":197,"wasted/sec":"4.2MB","cache":"33.0MB","delivered":5925,"sys":"9.2GB","alloc":"6.8GB","remaining":157118,"block_num":66055082,"t":"2025-01-02T09:10:15.565754153Z"}
{"cache":"10.1MB","delivered":6798,"msg":"[5/15 Bodies] Downloading block bodies","block_num":66056352,"t":"2025-01-02T09:10:45.56578616Z","blk/sec":226,"sys":"9.5GB","wasted/sec":"4.4MB","delivery/sec":"1.8MB","alloc":"4.1GB","remaining":155848,"lvl":"info"}
{"t":"2025-01-02T09:11:15.565359886Z","block_num":66057017,"delivery/sec":"1.8MB","lvl":"info","remaining":155183,"alloc":"5.0GB","blk/sec":256,"cache":"24.5MB","delivered":7706,"sys":"9.5GB","msg":"[5/15 Bodies] Downloading block bodies","wasted/sec":"4.2MB"}
{"blk/sec":283,"lvl":"info","sys":"9.5GB","alloc":"5.9GB","block_num":66057877,"delivery/sec":"1.5MB","cache":"19.9MB","wasted/sec":"4.3MB","remaining":154323,"t":"2025-01-02T09:11:45.56611209Z","delivered":8514,"msg":"[5/15 Bodies] Downloading block bodies"}
{"blk/sec":312,"alloc":"6.8GB","delivered":9369,"msg":"[5/15 Bodies] Downloading block bodies","remaining":153621,"t":"2025-01-02T09:12:15.565769235Z","block_num":66058594,"lvl":"info","wasted/sec":"4.4MB","delivery/sec":"1.7MB","cache":"29.4MB","sys":"9.5GB"}
{"wasted/sec":"4.3MB","cache":"26.9MB","block_num":66059523,"delivery/sec":"1.8MB","delivered":10237,"t":"2025-01-02T09:12:45.566399215Z","lvl":"info","blk/sec":341,"msg":"[5/15 Bodies] Downloading block bodies","sys":"9.5GB","remaining":152677,"alloc":"4.1GB"}
{"msg":"[5/15 Bodies] Downloading block bodies","delivered":11090,"wasted/sec":"4.3MB","lvl":"info","remaining":152035,"delivery/sec":"1.6MB","cache":"34.7MB","alloc":"5.0GB","block_num":66060165,"sys":"9.5GB","blk/sec":369,"t":"2025-01-02T09:13:15.565750397Z"}
{"sys":"9.5GB","alloc":"5.9GB","t":"2025-01-02T09:13:45.565887137Z","cache":"36.4MB","remaining":151084,"delivered":12012,"lvl":"info","msg":"[5/15 Bodies] Downloading block bodies","blk/sec":400,"delivery/sec":"1.9MB","wasted/sec":"4.0MB","block_num":66061116}
{"t":"2025-01-02T09:14:15.565486822Z","alloc":"6.8GB","blk/sec":427,"cache":"42.6MB","lvl":"info","block_num":66061923,"remaining":150277,"delivered":12834,"wasted/sec":"4.3MB","delivery/sec":"1.8MB","msg":"[5/15 Bodies] Downloading block bodies","sys":"9.5GB"}
{"block_num":66063026,"delivered":13806,"blk/sec":460,"msg":"[5/15 Bodies] Downloading block bodies","wasted/sec":"3.9MB","t":"2025-01-02T09:14:45.565955216Z","alloc":"4.1GB","delivery/sec":"2.2MB","remaining":149174,"sys":"9.5GB","cache":"34.9MB","lvl":"info"}
{"remaining":148506,"msg":"[5/15 Bodies] Downloading block bodies","block_num":66063736,"delivered":14542,"t":"2025-01-02T09:15:15.565852113Z","cache":"36.7MB","alloc":"5.0GB","wasted/sec":"4.1MB","delivery/sec":"1.7MB","sys":"9.5GB","lvl":"info","blk/sec":484}
{"wasted/sec":"4.0MB","remaining":147673,"alloc":"5.9GB","lvl":"info","delivery/sec":"2.1MB","msg":"[5/15 Bodies] Downloading block bodies","block_num":66064527,"blk/sec":514,"sys":"9.5GB","cache":"43.1MB","delivered":15436,"t":"2025-01-02T09:15:45.566485941Z"}
{"blk/sec":543,"remaining":146692,"sys":"9.5GB","t":"2025-01-02T09:16:15.565946591Z","alloc":"6.7GB","lvl":"info","msg":"[5/15 Bodies] Downloading block bodies","block_num":66065508,"cache":"37.6MB","wasted/sec":"4.0MB","delivered":16308,"delivery/sec":"2.1MB"}
{"alloc":"4.1GB","sys":"9.5GB","wasted/sec":"4.4MB","msg":"[5/15 Bodies] Downloading block bodies","delivery/sec":"1.6MB","delivered":17014,"cache":"31.9MB","lvl":"info","remaining":145927,"block_num":66066273,"t":"2025-01-02T09:16:45.56581755Z","blk/sec":567}
{"block_num":66067051,"wasted/sec":"3.8MB","delivered":17938,"remaining":145149,"sys":"9.5GB","blk/sec":597,"delivery/sec":"2.1MB","t":"2025-01-02T09:17:15.565214716Z","alloc":"5.0GB","lvl":"info","msg":"[5/15 Bodies] Downloading block bodies","cache":"40.9MB"}
{"remaining":144110,"alloc":"5.9GB","blk/sec":624,"wasted/sec":"4.3MB","delivery/sec":"1.8MB","msg":"[5/15 Bodies] Downloading block bodies","block_num":66068090,"sys":"9.5GB","cache":"25.2MB","lvl":"info","delivered":18742,"t":"2025-01-02T09:17:45.56589358Z"}
{"remaining":143180,"delivered":19655,"msg":"[5/15 Bodies] Downloading block bodies","lvl":"info","block_num":66069117,"t":"2025-01-02T09:18:15.566502992Z","blk/sec":655,"delivery/sec":"2.1MB","alloc":"6.8GB","wasted/sec":"3.9MB","sys":"9.5GB","cache":"17.8MB"}
{"delivered":20465,"cache":"23.2MB","msg":"[5/15 Bodies] Downloading block bodies","block_num":66069826,"t":"2025-01-02T09:18:45.566155702Z","lvl":"info","delivery/sec":"1.8MB","blk/sec":682,"wasted/sec":"4.1MB","sys":"9.5GB","alloc":"4.1GB","remaining":142374}
{"block_num":66070368,"lvl":"info","wasted/sec":"4.4MB","delivery/sec":"1.7MB","delivered":21279,"msg":"[5/15 Bodies] Downloading block bodies","cache":"39.6MB","remaining":141832,"blk/sec":709,"t":"2025-01-02T09:19:15.565743933Z","alloc":"5.0GB","sys":"9.5GB"}
{"delivered":22145,"blk/sec":738,"sys":"9.5GB","t":"2025-01-02T09:19:45.565768992Z","msg":"[5/15 Bodies] Downloading block bodies","block_num":66071326,"lvl":"info","wasted/sec":"4.1MB","remaining":140874,"delivery/sec":"2.0MB","alloc":"5.9GB","cache":"37.3MB"}

MrFreezeDZ avatar Jan 03 '25 11:01 MrFreezeDZ

We wondered why the disk space gets used a lot more in the last month. I found these log lines very interesting. We see an increase in "ReclaimableSpace" by a thousand times in one month. It looks like every restart during the standing still unwind process resulted in more space used.

2024-12-07T12:00:00.983597685Z {"AccountChangeSet":"148.9GB", "BlockTransaction":"330.4GB", "FreeList":"160.0KB", "PlainState":"479.2GB", "ReclaimableSpace":"640.0MB", "StorageChangeSet":"451.6GB", "TransactionLog":"2.2TB", "lvl":"info", "msg":"Tables", "t":"2024-12-07T12:00:00.983527115Z"}
2024-12-07T12:00:03.001582303Z {"AccountChangeSet":"148.9GB", "BlockTransaction":"330.4GB", "FreeList":"176.0KB", "PlainState":"479.2GB", "ReclaimableSpace":"704.0MB", "StorageChangeSet":"451.6GB", "TransactionLog":"2.2TB", "lvl":"info", "msg":"Tables", "t":"2024-12-07T12:00:03.001449173Z"}
2024-12-07T12:00:05.127530151Z {"AccountChangeSet":"148.9GB", "BlockTransaction":"330.4GB", "FreeList":"160.0KB", "PlainState":"479.2GB", "ReclaimableSpace":"640.0MB", "StorageChangeSet":"451.6GB", "TransactionLog":"2.2TB", "lvl":"info", "msg":"Tables", "t":"2024-12-07T12:00:05.127428301Z"}
2024-12-07T12:00:07.386769090Z {"AccountChangeSet":"148.9GB", "BlockTransaction":"330.4GB", "FreeList":"160.0KB", "PlainState":"479.2GB", "ReclaimableSpace":"640.0MB", "StorageChangeSet":"451.6GB", "TransactionLog":"2.2TB", "lvl":"info", "msg":"Tables", "t":"2024-12-07T12:00:07.38668852Z"}
2024-12-28T12:00:06.897921664Z {"AccountChangeSet":"155.3GB", "BlockTransaction":"386.5GB", "FreeList":"64.4MB", "PlainState":"484.7GB", "ReclaimableSpace":"257.6GB", "StorageChangeSet":"474.8GB", "TransactionLog":"2.3TB", "lvl":"info", "msg":"Tables", "t":"2024-12-28T12:00:06.897793384Z"}
2024-12-28T12:00:25.892152578Z {"AccountChangeSet":"155.3GB", "BlockTransaction":"386.5GB", "FreeList":"64.4MB", "PlainState":"484.7GB", "ReclaimableSpace":"257.6GB", "StorageChangeSet":"474.8GB", "TransactionLog":"2.3TB", "lvl":"info", "msg":"Tables", "t":"2024-12-28T12:00:25.892036888Z"}
2024-12-28T12:00:44.005144727Z {"AccountChangeSet":"155.3GB", "BlockTransaction":"386.5GB", "FreeList":"64.4MB", "PlainState":"484.7GB", "ReclaimableSpace":"257.6GB", "StorageChangeSet":"474.8GB", "TransactionLog":"2.3TB", "lvl":"info", "msg":"Tables", "t":"2024-12-28T12:00:44.004967377Z"}
2025-01-03T20:55:55.899760900Z {"AccountChangeSet":"156.7GB", "BlockTransaction":"399.3GB", "FreeList":"62.3MB", "PlainState":"486.4GB", "ReclaimableSpace":"249.2GB", "StorageChangeSet":"479.9GB", "TransactionLog":"2.3TB", "lvl":"info", "msg":"Tables", "t":"2025-01-03T20:55:55.89965635Z"}
2025-01-04T12:06:18.388980857Z {"AccountChangeSet":"157.2GB", "BlockTransaction":"403.3GB", "FreeList":"172.4MB", "PlainState":"487.0GB", "ReclaimableSpace":"689.6GB", "StorageChangeSet":"481.5GB", "TransactionLog":"2.3TB", "lvl":"info", "msg":"Tables", "t":"2025-01-04T12:06:18.388820517Z"}
2025-01-04T20:54:40.343019100Z {"AccountChangeSet":"157.4GB", "BlockTransaction":"405.1GB", "FreeList":"167.0MB", "PlainState":"487.2GB", "ReclaimableSpace":"667.9GB", "StorageChangeSet":"482.2GB", "TransactionLog":"2.3TB", "lvl":"info", "msg":"Tables", "t":"2025-01-04T20:54:40.3429199Z"}
2025-01-05T03:05:59.639916743Z {"AccountChangeSet":"157.5GB", "BlockTransaction":"406.0GB", "FreeList":"164.7MB", "PlainState":"487.4GB", "ReclaimableSpace":"658.9GB", "StorageChangeSet":"482.6GB", "TransactionLog":"2.3TB", "lvl":"info", "msg":"Tables", "t":"2025-01-05T03:05:59.639779263Z"}
2025-01-05T06:51:13.395888601Z {"AccountChangeSet":"157.6GB", "BlockTransaction":"406.7GB", "FreeList":"163.4MB", "PlainState":"487.4GB", "ReclaimableSpace":"653.7GB", "StorageChangeSet":"482.9GB", "TransactionLog":"2.3TB", "lvl":"info", "msg":"Tables", "t":"2025-01-05T06:51:13.395849111Z"}
2025-01-05T10:24:03.005198100Z {"AccountChangeSet":"157.6GB", "BlockTransaction":"407.1GB", "FreeList":"162.7MB", "PlainState":"487.5GB", "ReclaimableSpace":"650.7GB", "StorageChangeSet":"483.1GB", "TransactionLog":"2.3TB", "lvl":"info", "msg":"Tables", "t":"2025-01-05T10:24:03.00506535Z"}

Could this be related to this issue here in some way?

MrFreezeDZ avatar Jan 06 '25 14:01 MrFreezeDZ

Here we have the current used space of the data folder of Erigon:

du -h /data
16.0K   /data/lost+found
362.2M  /data/logs
40.0M   /data/nodes/eth67
37.0M   /data/nodes/eth68
77.0M   /data/nodes
4.0K    /data/snapshots/idx
4.0K    /data/snapshots/accessor
4.0K    /data/snapshots/domain
4.0K    /data/snapshots/history
1.1T    /data/snapshots
4.0K    /data/caplin/blobs
4.0K    /data/caplin/indexing
12.0K   /data/caplin
5.8T    /data/chaindata
1.7M    /data/diagnostics
49.0M   /data/downloader
128.6M  /data/bor
36.0K   /data/temp
8.5G    /data/txpool
7.0T    /data

ls -larth /data/txpool
total 9G     
drwxrwsr--    2 erigon   erigon      4.0K Dec 15  2023 .
drwxrwsr-x   13 root     erigon      4.0K Jul 15 09:26 ..
-rw-rw-r--    1 erigon   erigon   1004.0K Jan  9 09:35 mdbx.lck
-rw-rw-r--    1 erigon   erigon      8.5G Jan  9 09:35 mdbx.dat

We wonder why the txpool is this big.

MrFreezeDZ avatar Jan 09 '25 09:01 MrFreezeDZ

Please re-open if the issue happens with Erigon v3.0.2 or later.

yperbasis avatar May 05 '25 09:05 yperbasis