erigon icon indicating copy to clipboard operation
erigon copied to clipboard
verified publisher icon erigontech

Polygon Erigon Syncing From Scratch stalls at block 48077378 across 5 nodes

Open tobidae-cb opened this issue 1 year ago • 6 comments

System information

Erigon version: v2.60.1-9471c44e

OS & Version: Linux

Erigon Command (with flags/config):

    EXECUTION_BOOT_NODES="${EXECUTION_BOOT_NODES:-}"
    KEEP_LAST_BLOCKS="${KEEP_LAST_BLOCKS:-2600000}"
    HEIMDALL_URL="http://localhost:1317"

    echo "Starting erigon client for polygon"
    exec /app/erigon \
    --chain "${CHAIN}" \
    --bor.heimdall "${HEIMDALL_URL}" \
    --rpc.txfeecap=0 \
    --http \
    --http.addr 0.0.0.0 \
    --http.vhosts="*" \
    --http.api="engine,eth,web3,net,debug,trace,txpool,erigon,admin,bor" \
    --http.port "${EXECUTION_HTTP_PORT}" \
    --http.corsdomain="*" \
    --http.trace="true" \
    --port "${PEERING_PORT}" \
    --datadir "${DATA_DIR}" \
    --ws \
    --ws.port "${WEBSOCKET_PORT}" \
    --metrics \
    --metrics.addr 0.0.0.0 \
    --private.api.addr 0.0.0.0:9091 \
    --metrics.addr 0.0.0.0 \
    --metrics.port 6062 \
    --pprof \
    --pprof.addr 0.0.0.0 \
    --pprof.port 6063 \
    --bootnodes "${EXECUTION_BOOT_NODES}" \
    --torrent.download.rate=120mb \
    --torrent.staticpeers="${EXECUTION_BOOT_NODES}" \
    --state.cache=16384 \
    --rpc.batch.limit=100000 \
    --torrent.download.slots=6 \
    --torrent.verbosity="4" \
    --batchSize=512M \
    --prune.h.older "${KEEP_LAST_BLOCKS}" \
    --prune.r.older "${KEEP_LAST_BLOCKS}" \
    --prune.t.older "${KEEP_LAST_BLOCKS}" \
    --prune.c.older "${KEEP_LAST_BLOCKS}"

Consensus Layer: Heimdall

Consensus Layer Command (with flags/config):

##### RPC and REST configs #####

# RPC endpoint for ethereum chain
eth_rpc_url = "http://127.0.0.1:9545"

# RPC endpoint for bor chain
bor_rpc_url = "http://127.0.0.1:8545"

# RPC endpoint for tendermint
tendermint_rpc_url = "http://0.0.0.0:26657"

# Heimdall REST server endpoint
heimdall_rest_server = "http://0.0.0.0:1317"

#### Bridge configs ####

# AMQP endpoint
amqp_url = "amqp://guest:[email protected]:5672/"

## Poll intervals
checkpoint_poll_interval = "5m0s"
syncer_poll_interval = "1m0s"
noack_poll_interval = "16m50s"
clerk_poll_interval = "10s"
span_poll_interval = "1m0s"

#### gas limits ####
main_chain_gas_limit = "5000000"

#### gas price ####
main_chain_max_gas_price = "400000000000"

##### Timeout Config #####
no_ack_wait_time = "30m0s"

Chain/Network: Mainnet

Expected behaviour

The nodes should be syncing to tip over time.

Actual behaviour

The nodes appear to be stalled at block 48077378 even with 18 peers.

Steps to reproduce the behaviour

Sync mainnet node from scratch with the configuration provided above.

Backtrace

The logs below keep getting printed for the last 4 days now

polygon-0 erigon [INFO] [06-24|15:33:33.224] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=6.8GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:33:38.293] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=6.7GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:34:00.856] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=7.7GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:34:34.962] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=5.1GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:35:08.855] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=6.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:35:24.853] [p2p] GoodPeers                          eth68=12 eth66=1 eth67=5
polygon-0 erigon [INFO] [06-24|15:35:26.340] [mem] memory stats                       Rss=80.3GB Size=0B Pss=80.3GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.1GB PrivateDirty=10.2GB Referenced=80.3GB Anonymous=10.2GB Swap=0B alloc=7.8GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:35:31.376] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=7.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:36:05.488] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=5.1GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:36:33.223] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=5.9GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:36:39.606] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=6.8GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:37:02.248] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=7.8GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:37:36.395] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=5.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:38:10.675] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=6.7GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:38:24.853] [p2p] GoodPeers                          eth66=1 eth68=12 eth67=5
polygon-0 erigon [INFO] [06-24|15:38:26.330] [mem] memory stats                       Rss=80.3GB Size=0B Pss=80.3GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.1GB PrivateDirty=10.2GB Referenced=80.3GB Anonymous=10.2GB Swap=0B alloc=6.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:38:33.634] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=7.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:39:07.901] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=5.0GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:39:30.476] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=6.0GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:39:33.223] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=8.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:40:04.450] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=7.5GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:40:38.507] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=5.1GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:41:01.243] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=6.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:41:24.853] [p2p] GoodPeers                          eth68=12 eth67=5 eth66=1
polygon-0 erigon [INFO] [06-24|15:41:26.424] [mem] memory stats                       Rss=80.3GB Size=0B Pss=80.3GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.1GB PrivateDirty=10.2GB Referenced=80.3GB Anonymous=10.2GB Swap=0B alloc=4.9GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:41:35.402] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=7.7GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:42:09.471] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=5.3GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:42:32.306] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=6.4GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:42:33.223] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=7.0GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:43:06.382] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=7.9GB sys=9.7GB

tobidae-cb avatar Jun 24 '24 15:06 tobidae-cb

grep -v Writing block bodies | grep -v p2p | grep -v txpool

AskAlexSharov avatar Jun 25 '24 03:06 AskAlexSharov

I'm assuming you want to see the non-memory stats info but just in case

+ polygon-0 › erigon
polygon-0 erigon [INFO] [06-23|17:18:00.149] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=108.6MB wasted/sec=35.1KB remaining=1889576 delivered=2022152 blk/sec=67405 cache=256.0MB alloc=4.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-23|19:13:00.149] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=41.7MB wasted/sec=35.1KB remaining=1889576 delivered=2047121 blk/sec=68237 cache=256.0MB alloc=5.1GB sys=9.7GB
polygon-0 erigon meta_checktxnid:11415 catch invalid root_page_txnid 4710637 for maindb.mod_txnid 4710634 (workaround for incoherent flaw of unified page/buffer cache)
polygon-0 erigon [INFO] [06-24|00:40:30.149] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=119.1MB wasted/sec=0B remaining=1889576 delivered=2118420 blk/sec=70614 cache=256.0MB alloc=7.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|06:57:00.150] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=136.5MB wasted/sec=70.1KB remaining=1889576 delivered=2200133 blk/sec=73337 cache=256.0MB alloc=6.3GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|07:54:30.150] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=20.8MB wasted/sec=0B remaining=1889576 delivered=2212597 blk/sec=73753 cache=256.0MB alloc=8.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|11:46:00.149] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=83.8MB wasted/sec=0B remaining=1889576 delivered=2262781 blk/sec=75426 cache=256.0MB alloc=7.7GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|13:16:00.150] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=32.7MB wasted/sec=0B remaining=1889576 delivered=2282379 blk/sec=76079 cache=256.0MB alloc=7.4GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|16:46:00.153] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=76.2MB wasted/sec=35.1KB remaining=1889576 delivered=2327971 blk/sec=77599 cache=256.0MB alloc=7.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|22:36:30.150] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=127.1MB wasted/sec=105.2KB remaining=1889576 delivered=2403985 blk/sec=80132 cache=256.0MB alloc=7.7GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|23:43:00.149] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=24.1MB wasted/sec=35.1KB remaining=1889576 delivered=2418417 blk/sec=80613 cache=256.0MB alloc=8.2GB sys=9.7GB

polygon-0 erigon [INFO] [06-25|13:20:26.462] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=4.9GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:23:26.404] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=8.3GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:25:30.149] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=298.2MB wasted/sec=70.1KB remaining=1889576 delivered=2596890 blk/sec=86563 cache=256.0MB alloc=7.9GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:26:26.354] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:29:26.353] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.0GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:32:26.448] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=5.0GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:35:26.493] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=4.7GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:38:26.352] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=8.1GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:41:26.355] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=7.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:44:26.355] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=7.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:47:26.366] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:50:26.360] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.0GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:53:26.431] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=5.1GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:56:26.422] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.3GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:59:26.357] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=8.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:02:26.356] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=7.5GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:05:26.348] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.8GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:08:26.369] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.4GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:11:26.365] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.0GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:14:26.447] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=4.9GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:17:26.437] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=7.3GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:20:26.357] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=5.7GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:23:26.423] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=5.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:26:26.341] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=8.3GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:29:26.347] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=7.5GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:32:26.373] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.8GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:35:26.364] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=5.4GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:38:26.368] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=7.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:41:26.344] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=5.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:44:26.439] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=4.9GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:47:26.342] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:50:26.374] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=8.0GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:53:26.355] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.4GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:56:26.457] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=5.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:59:26.344] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=7.1GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|15:02:26.437] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=4.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|15:05:26.346] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=7.4GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|15:08:26.351] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.4GB sys=9.7GB

tobidae-cb avatar Jun 25 '24 15:06 tobidae-cb

Here's a different pod

polygon-2 erigon [INFO] [06-25|15:37:10.858] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:12.451] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:12.452] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:13.439] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:15.062] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:15.062] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:16.043] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:17.646] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:17.646] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:18.626] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:20.257] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:20.257] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:21.241] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:22.824] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:22.824] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:23.803] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:25.405] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:25.405] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:26.401] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:28.016] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:28.016] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:28.998] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:30.607] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:30.607] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:31.587] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:33.223] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:33.223] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:34.197] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:35.802] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:35.802] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:36.776] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:38.387] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424

tobidae-cb avatar Jun 25 '24 15:06 tobidae-cb

Other pods

+ polygon-1 › erigon
polygon-1 erigon meta_checktxnid:11415 catch invalid root_page_txnid 4676552 for maindb.mod_txnid 4676547 (workaround for incoherent flaw of unified page/buffer cache)
polygon-1 erigon [INFO] [06-24|14:14:00.389] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=1.0GB wasted/sec=35.1KB remaining=1889343 delivered=3120005 blk/sec=104000 cache=256.0MB alloc=5.2GB sys=10.3GB
polygon-1 erigon [INFO] [06-24|17:31:30.390] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=107.4MB wasted/sec=35.1KB remaining=1889343 delivered=3174815 blk/sec=105827 cache=256.0MB alloc=8.1GB sys=10.3GB
polygon-1 erigon meta_checktxnid:11415 catch invalid root_page_txnid 4883934 for maindb.mod_txnid 4883928 (workaround for incoherent flaw of unified page/buffer cache)
polygon-1 erigon [INFO] [06-25|07:54:30.389] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=469.0MB wasted/sec=35.1KB remaining=1889343 delivered=3414215 blk/sec=113807 cache=256.0MB alloc=5.1GB sys=10.3GB

+ polygon-3 › erigon
polygon-3 erigon [INFO] [06-24|00:50:41.009] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=835.0MB wasted/sec=18.1MB remaining=1889041 delivered=3674616 blk/sec=122487 cache=256.0MB alloc=8.0GB sys=9.5GB
polygon-3 erigon [INFO] [06-24|05:38:11.010] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=207.3MB wasted/sec=34.2KB remaining=1889041 delivered=3776883 blk/sec=125896 cache=256.0MB alloc=8.1GB sys=9.5GB
polygon-3 erigon [INFO] [06-24|07:08:11.009] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=64.8MB wasted/sec=34.2KB remaining=1889041 delivered=3808827 blk/sec=126960 cache=256.0MB alloc=5.8GB sys=9.5GB
polygon-3 erigon [INFO] [06-24|18:24:11.009] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=486.8MB wasted/sec=68.5KB remaining=1889041 delivered=4048968 blk/sec=134965 cache=256.0MB alloc=4.6GB sys=9.5GB
polygon-3 erigon [INFO] [06-24|20:48:11.010] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=103.7MB wasted/sec=0B remaining=1889041 delivered=4100118 blk/sec=136670 cache=256.0MB alloc=4.6GB sys=9.5GB
polygon-3 erigon [INFO] [06-25|02:28:41.009] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=245.6MB wasted/sec=34.2KB remaining=1889041 delivered=4221294 blk/sec=140709 cache=256.0MB alloc=6.2GB sys=9.5GB
polygon-3 erigon [INFO] [06-25|03:38:11.010] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=50.1MB wasted/sec=34.2KB remaining=1889041 delivered=4245978 blk/sec=141532 cache=256.0MB alloc=6.5GB sys=9.5GB
polygon-3 erigon [INFO] [06-25|07:50:41.010] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=181.6MB wasted/sec=0B remaining=1889041 delivered=4335573 blk/sec=144519 cache=256.0MB alloc=7.2GB sys=9.5GB

tobidae-cb avatar Jun 25 '24 15:06 tobidae-cb

@AskAlexSharov thoughts on the potential issue?

tobidae-cb avatar Jun 26 '24 17:06 tobidae-cb

try integration stage_headers --reset

AskAlexSharov avatar Jun 27 '24 04:06 AskAlexSharov

Hi this is what I got after running the integration stage_headers --reset command.

+ polygon-3 › erigon-integration
polygon-3 erigon-integration + /app/integration --help
polygon-3 erigon-integration time="2024-07-09T06:08:47Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/cpu.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:08:47Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/memory.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:08:47Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/io.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:08:47Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/cpu.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:08:47Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/memory.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:08:47Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/io.pressure: operation not supported"
polygon-3 erigon-integration long and heavy integration tests for Erigon
polygon-3 erigon-integration
polygon-3 erigon-integration Usage:
polygon-3 erigon-integration   integration [command]
polygon-3 erigon-integration
polygon-3 erigon-integration Available Commands:
polygon-3 erigon-integration   clear_bad_blocks     Clear table with bad block hashes to allow to process this blocks one more time
polygon-3 erigon-integration   compare_bucket       compare bucket to the same bucket in '--chaindata.reference'
polygon-3 erigon-integration   compare_states       compare state buckets to buckets in '--chaindata.reference'
polygon-3 erigon-integration   completion           Generate the autocompletion script for the specified shell
polygon-3 erigon-integration   f_to_mdbx            copy data from '--chaindata' to '--chaindata.to'
polygon-3 erigon-integration   force_set_history_v3 Override existing --history.v3 flag value (if you know what you are doing)
polygon-3 erigon-integration   force_set_prune      Override existing --prune flag value (if you know what you are doing)
polygon-3 erigon-integration   force_set_snap       Override existing --snapshots flag value (if you know what you are doing)
polygon-3 erigon-integration   help                 Help about any command
polygon-3 erigon-integration   loop_exec
polygon-3 erigon-integration   loop_ih
polygon-3 erigon-integration   mdbx_to_mdbx         copy data from '--chaindata' to '--chaindata.to'
polygon-3 erigon-integration   print_migrations
polygon-3 erigon-integration   print_stages
polygon-3 erigon-integration   print_table_sizes
polygon-3 erigon-integration   remove_migration
polygon-3 erigon-integration   reset_state          Reset StateStages (5,6,7,8,9,10) and buckets
polygon-3 erigon-integration   run_migrations
polygon-3 erigon-integration   stage_bodies
polygon-3 erigon-integration   stage_bor_heimdall
polygon-3 erigon-integration   stage_call_traces
polygon-3 erigon-integration   stage_exec
polygon-3 erigon-integration   stage_hash_state
polygon-3 erigon-integration   stage_headers
polygon-3 erigon-integration   stage_history
polygon-3 erigon-integration   stage_log_index
polygon-3 erigon-integration   stage_senders
polygon-3 erigon-integration   stage_snapshots
polygon-3 erigon-integration   stage_trie
polygon-3 erigon-integration   stage_tx_lookup
polygon-3 erigon-integration   state_stages         Run all StateStages (which happen after senders) in loop.
polygon-3 erigon-integration Examples:
polygon-3 erigon-integration --unwind=1 --unwind.every=10  # 10 blocks forward, 1 block back, 10 blocks forward, ...
polygon-3 erigon-integration --unwind=10 --unwind.every=1  # 1 block forward, 10 blocks back, 1 blocks forward, ...
polygon-3 erigon-integration --unwind=10  # 10 blocks back, then stop
polygon-3 erigon-integration --integrity.fast=false --integrity.slow=false # Performs DB integrity checks each step. You can disable slow or fast checks.
polygon-3 erigon-integration --block # Stop at exact blocks
polygon-3 erigon-integration --chaindata.reference # When finish all cycles, does comparison to this db file.
polygon-3 erigon-integration
polygon-3 erigon-integration   warmup
polygon-3 erigon-integration
polygon-3 erigon-integration Flags:
polygon-3 erigon-integration       --diagnostics.disabled               Disable diagnostics
polygon-3 erigon-integration       --diagnostics.endpoint.addr string   Diagnostics HTTP server listening interface (default "0.0.0.0")
polygon-3 erigon-integration       --diagnostics.endpoint.port uint     Diagnostics HTTP server listening port (default 6060)
polygon-3 erigon-integration       --diagnostics.speedtest              Enable speed test
polygon-3 erigon-integration   -h, --help                               help for integration
polygon-3 erigon-integration       --log.console.json                   Format console logs with JSON
polygon-3 erigon-integration       --log.console.verbosity string       Set the log level for console logs (default "info")
polygon-3 erigon-integration       --log.delays                         Enable block delay logging
polygon-3 erigon-integration       --log.dir.disable                    disable disk logging
polygon-3 erigon-integration       --log.dir.json                       Format file logs with JSON
polygon-3 erigon-integration       --log.dir.path string                Path to store user and error logs to disk
polygon-3 erigon-integration       --log.dir.prefix string              The file name prefix for logs stored to disk
polygon-3 erigon-integration       --log.dir.verbosity string           Set the log verbosity for logs stored to disk (default "info")
polygon-3 erigon-integration       --log.json                           Format console logs with JSON
polygon-3 erigon-integration       --metrics                            Enable metrics collection and reporting
polygon-3 erigon-integration       --metrics.addr string                Enable stand-alone metrics HTTP server listening interface (default "0.0.0.0")
polygon-3 erigon-integration       --metrics.port int                   Metrics HTTP server listening port (default 6060)
polygon-3 erigon-integration       --pprof                              Enable the pprof HTTP server
polygon-3 erigon-integration       --pprof.addr string                  pprof HTTP server listening interface (default "0.0.0.0")
polygon-3 erigon-integration       --pprof.cpuprofile string            Write CPU profile to the given file
polygon-3 erigon-integration       --pprof.port int                     pprof HTTP server listening port (default 6060)
polygon-3 erigon-integration       --trace string                       Write execution trace to the given file
polygon-3 erigon-integration       --verbosity string                   Set the log level for console logs (default "info")
polygon-3 erigon-integration
polygon-3 erigon-integration Use "integration [command] --help" for more information about a command.
polygon-3 erigon-integration + echo 'Printing erigon stages before reset'
polygon-3 erigon-integration Printing erigon stages before reset
polygon-3 erigon-integration + /app/integration print_stages --datadir /data/erigon --chain bor-mainnet
polygon-3 erigon-integration time="2024-07-09T06:08:48Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/cpu.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:08:48Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/memory.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:08:48Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/io.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:08:48Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/cpu.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:08:48Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/memory.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:08:48Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/io.pressure: operation not supported"
polygon-3 erigon-integration [INFO] [07-09|06:08:48.191] logging to file system                   log dir=/data/erigon/logs file prefix=integration log level=info json=false
polygon-3 erigon-integration time="2024-07-09T06:08:48Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/cpu.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:08:48Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/memory.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:08:48Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/io.pressure: operation not supported"
polygon-3 erigon-integration [INFO] [07-09|06:08:48.195] [db] open                                label=chaindata sizeLimit=12TB pageSize=8192
polygon-3 erigon-integration [INFO] [07-09|06:08:54.573] [snapshots:all] Stat                     blocks=56300k indices=56300k alloc=2.9GB sys=3.3GB
polygon-3 erigon-integration Note: prune_at doesn't mean 'all data before were deleted' - it just mean stage.Prune function were run to this block. Because 1 stage may prune multiple data types to different prune distance.
polygon-3 erigon-integration
polygon-3 erigon-integration [INFO] [07-09|06:08:54.912] [snapshots:all] Stat                     blocks=54600k indices=54600k alloc=3.0GB sys=3.3GB
polygon-3 erigon-integration  				 stage_at 	 prune_at
polygon-3 erigon-integration Snapshots 			 56299999 	 0
polygon-3 erigon-integration Headers 			 58189058 	 0
polygon-3 erigon-integration BorHeimdall 			 58189058 	 0
polygon-3 erigon-integration BlockHashes 			 58189058 	 0
polygon-3 erigon-integration Bodies 				 56300016 	 0
polygon-3 erigon-integration Senders 			 56299999 	 0
polygon-3 erigon-integration Execution 			 48077378 	 0
polygon-3 erigon-integration Translation 			 0 		 0
polygon-3 erigon-integration HashState 			 0 		 0
polygon-3 erigon-integration IntermediateHashes 		 0 		 0
polygon-3 erigon-integration AccountHistoryIndex 		 0 		 0
polygon-3 erigon-integration StorageHistoryIndex 		 0 		 0
polygon-3 erigon-integration LogIndex 			 0 		 0
polygon-3 erigon-integration CallTraces 			 0 		 0
polygon-3 erigon-integration TxLookup 			 0 		 0
polygon-3 erigon-integration Finish 				 0 		 0
polygon-3 erigon-integration --
polygon-3 erigon-integration prune distance: --prune.h.older=2600000 --prune.r.older=2600000 --prune.t.older=2600000 --prune.c.older=2600000
polygon-3 erigon-integration
polygon-3 erigon-integration blocks.v2: true, blocks=56299999, segments=56299999, indices=56299999
polygon-3 erigon-integration blocks.bor.v2: segments=54599999, indices=54599999
polygon-3 erigon-integration
polygon-3 erigon-integration history.v3: false,  idx steps: 0.00, lastBlockInSnap=0, TxNums_Index(0,1)
polygon-3 erigon-integration
polygon-3 erigon-integration sequence: EthTx=4030064003, NonCanonicalTx=0
polygon-3 erigon-integration
polygon-3 erigon-integration in db: first header 56300000, last header 58189058, first body 56300000, last body 56316015
polygon-3 erigon-integration --
polygon-3 erigon-integration + echo 'Reset erigon stages'
polygon-3 erigon-integration + /app/integration stage_headers --reset --datadir /data/erigon --chain bor-mainnet
polygon-3 erigon-integration Reset erigon stages
polygon-3 erigon-integration time="2024-07-09T06:08:55Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/cpu.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:08:55Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/memory.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:08:55Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/io.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:08:55Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/cpu.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:08:55Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/memory.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:08:55Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/io.pressure: operation not supported"
polygon-3 erigon-integration [INFO] [07-09|06:08:55.623] logging to file system                   log dir=/data/erigon/logs file prefix=integration log level=info json=false
polygon-3 erigon-integration time="2024-07-09T06:08:55Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/cpu.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:08:55Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/memory.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:08:55Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/io.pressure: operation not supported"
polygon-3 erigon-integration [INFO] [07-09|06:08:55.625] [db] open                                label=chaindata sizeLimit=12TB pageSize=8192
polygon-3 erigon-integration [INFO] [07-09|06:08:59.891] [snapshots:all] Stat                     blocks=56300k indices=56300k alloc=3.2GB sys=3.4GB
polygon-3 erigon-integration [INFO] [07-09|06:09:00.071] [snapshots:all] Stat                     blocks=54600k indices=54600k alloc=3.3GB sys=3.5GB
polygon-3 erigon-integration [INFO] [07-09|06:09:00.088] Opening Database                         label=bor path=/data/erigon/bor
polygon-3 erigon-integration time="2024-07-09T06:09:00Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/cpu.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:09:00Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/memory.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:09:00Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/io.pressure: operation not supported"
polygon-3 erigon-integration mdbx_setup_dxb:15946 filesize mismatch (expect 175677440b/21445p, have 184549376b/22528p)
polygon-3 erigon-integration [INFO] [07-09|06:09:22.235] TruncateBlocks                           block=56314009
polygon-3 erigon-integration [INFO] [07-09|06:09:42.231] TruncateBlocks                           block=57525611
polygon-3 erigon-integration [INFO] [07-09|06:11:55.827] [mem] memory stats                       Rss=10.5GB Size=0B Pss=10.5GB SharedClean=2.5MB SharedDirty=0B PrivateClean=6.0GB PrivateDirty=4.5GB Referenced=10.5GB Anonymous=4.5GB Swap=0B alloc=3.2GB sys=4.5GB
polygon-3 erigon-integration [INFO] [07-09|06:14:03.469] Clear                                    table=NonCanonicalTransaction
polygon-3 erigon-integration [INFO] [07-09|06:14:03.469] Clear                                    table=BlockTransaction
polygon-3 erigon-integration [INFO] [07-09|06:14:03.469] Clear                                    table=MaxTxNum
polygon-3 erigon-integration [INFO] [07-09|06:14:33.472] [filling_db_from_snapshots] Total difficulty index: 3526k/56299k
polygon-3 erigon-integration [INFO] [07-09|06:14:55.918] [mem] memory stats                       Rss=17.1GB Size=0B Pss=17.1GB SharedClean=2.5MB SharedDirty=0B PrivateClean=9.8GB PrivateDirty=7.3GB Referenced=17.1GB Anonymous=7.3GB Swap=0B alloc=3.2GB sys=6.8GB
polygon-3 erigon-integration [INFO] [07-09|06:15:03.472] [filling_db_from_snapshots] Total difficulty index: 6119k/56299k
polygon-3 erigon-integration [INFO] [07-09|06:15:33.472] [filling_db_from_snapshots] Total difficulty index: 8583k/56299k
polygon-3 erigon-integration [INFO] [07-09|06:16:06.146] [filling_db_from_snapshots] Total difficulty index: 11184k/56299k
polygon-3 erigon-integration [INFO] [07-09|06:16:33.472] [filling_db_from_snapshots] Total difficulty index: 13622k/56299k
polygon-3 erigon-integration [INFO] [07-09|06:17:03.475] [filling_db_from_snapshots] Total difficulty index: 16492k/56299k
polygon-3 erigon-integration [INFO] [07-09|06:17:33.472] [filling_db_from_snapshots] Total difficulty index: 19618k/56299k
polygon-3 erigon-integration [INFO] [07-09|06:17:56.014] [mem] memory stats                       Rss=24.6GB Size=0B Pss=24.6GB SharedClean=2.5MB SharedDirty=0B PrivateClean=15.6GB PrivateDirty=9.1GB Referenced=24.6GB Anonymous=8.8GB Swap=0B alloc=5.3GB sys=6.8GB
polygon-3 erigon-integration [INFO] [07-09|06:18:03.472] [filling_db_from_snapshots] Total difficulty index: 22790k/56299k
polygon-3 erigon-integration [INFO] [07-09|06:18:35.627] [filling_db_from_snapshots] Total difficulty index: 26097k/56299k
polygon-3 erigon-integration [INFO] [07-09|06:19:03.471] [filling_db_from_snapshots] Total difficulty index: 29772k/56299k
polygon-3 erigon-integration [INFO] [07-09|06:19:33.472] [filling_db_from_snapshots] Total difficulty index: 32771k/56299k
polygon-3 erigon-integration [INFO] [07-09|06:20:03.472] [filling_db_from_snapshots] Total difficulty index: 35786k/56299k
polygon-3 erigon-integration [INFO] [07-09|06:20:33.472] [filling_db_from_snapshots] Total difficulty index: 38776k/56299k
polygon-3 erigon-integration [INFO] [07-09|06:20:56.157] [mem] memory stats                       Rss=35.0GB Size=0B Pss=35.0GB SharedClean=2.5MB SharedDirty=0B PrivateClean=26.2GB PrivateDirty=8.8GB Referenced=35.0GB Anonymous=8.8GB Swap=0B alloc=5.0GB sys=6.8GB
polygon-3 erigon-integration [INFO] [07-09|06:21:03.478] [filling_db_from_snapshots] Total difficulty index: 41663k/56299k
polygon-3 erigon-integration [INFO] [07-09|06:21:34.927] [filling_db_from_snapshots] Total difficulty index: 44739k/56299k
polygon-3 erigon-integration [INFO] [07-09|06:22:03.472] [filling_db_from_snapshots] Total difficulty index: 48280k/56299k
polygon-3 erigon-integration [INFO] [07-09|06:22:33.472] [filling_db_from_snapshots] Total difficulty index: 51149k/56299k
polygon-3 erigon-integration [INFO] [07-09|06:23:03.472] [filling_db_from_snapshots] Total difficulty index: 54111k/56299k
polygon-3 erigon-integration [INFO] [07-09|06:23:56.292] [mem] memory stats                       Rss=44.5GB Size=0B Pss=44.5GB SharedClean=2.5MB SharedDirty=0B PrivateClean=35.7GB PrivateDirty=8.8GB Referenced=44.5GB Anonymous=8.8GB Swap=0B alloc=5.3GB sys=6.8GB
polygon-3 erigon-integration Printing erigon stages after reset
polygon-3 erigon-integration + echo 'Printing erigon stages after reset'
polygon-3 erigon-integration + /app/integration print_stages --datadir /data/erigon --chain bor-mainnet
polygon-3 erigon-integration time="2024-07-09T06:26:18Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/cpu.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:26:18Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/memory.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:26:18Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/io.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:26:18Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/cpu.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:26:18Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/memory.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:26:18Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/io.pressure: operation not supported"
polygon-3 erigon-integration [INFO] [07-09|06:26:18.241] logging to file system                   log dir=/data/erigon/logs file prefix=integration log level=info json=false
polygon-3 erigon-integration time="2024-07-09T06:26:18Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/cpu.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:26:18Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/memory.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:26:18Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/io.pressure: operation not supported"
polygon-3 erigon-integration [INFO] [07-09|06:26:18.243] [db] open                                label=chaindata sizeLimit=12TB pageSize=8192
polygon-3 erigon-integration [INFO] [07-09|06:26:22.641] [snapshots:all] Stat                     blocks=56300k indices=56300k alloc=3.2GB sys=3.4GB
polygon-3 erigon-integration [INFO] [07-09|06:26:22.822] [snapshots:all] Stat                     blocks=54600k indices=54600k alloc=3.3GB sys=3.5GB
polygon-3 erigon-integration Note: prune_at doesn't mean 'all data before were deleted' - it just mean stage.Prune function were run to this block. Because 1 stage may prune multiple data types to different prune distance.
polygon-3 erigon-integration
polygon-3 erigon-integration  				 stage_at 	 prune_at
polygon-3 erigon-integration Snapshots 			 56299999 	 0
polygon-3 erigon-integration Headers 			 56299999 	 0
polygon-3 erigon-integration BorHeimdall 			 58189058 	 0
polygon-3 erigon-integration BlockHashes 			 58189058 	 0
polygon-3 erigon-integration Bodies 				 56299999 	 0
polygon-3 erigon-integration Senders 			 56299999 	 0
polygon-3 erigon-integration Execution 			 48077378 	 0
polygon-3 erigon-integration Translation 			 0 		 0
polygon-3 erigon-integration HashState 			 0 		 0
polygon-3 erigon-integration IntermediateHashes 		 0 		 0
polygon-3 erigon-integration AccountHistoryIndex 		 0 		 0
polygon-3 erigon-integration StorageHistoryIndex 		 0 		 0
polygon-3 erigon-integration LogIndex 			 0 		 0
polygon-3 erigon-integration CallTraces 			 0 		 0
polygon-3 erigon-integration TxLookup 			 0 		 0
polygon-3 erigon-integration Finish 				 0 		 0
polygon-3 erigon-integration --
polygon-3 erigon-integration prune distance: --prune.h.older=2600000 --prune.r.older=2600000 --prune.t.older=2600000 --prune.c.older=2600000
polygon-3 erigon-integration
polygon-3 erigon-integration blocks.v2: true, blocks=56299999, segments=56299999, indices=56299999
polygon-3 erigon-integration blocks.bor.v2: segments=54599999, indices=54599999
polygon-3 erigon-integration
polygon-3 erigon-integration history.v3: false,  idx steps: 0.00, lastBlockInSnap=0, TxNums_Index(0,0)
polygon-3 erigon-integration
polygon-3 erigon-integration sequence: EthTx=4030064003, NonCanonicalTx=0
polygon-3 erigon-integration
polygon-3 erigon-integration in db: first header 0, last header 0, first body 0, last body 0
polygon-3 erigon-integration --
polygon-3 erigon-integration + echo 'Reset erigon stage log index since upgrading to 2.60.x'
polygon-3 erigon-integration + /app/integration stage_log_index --reset --datadir=/data/erigon --chain=bor-mainnet
polygon-3 erigon-integration Reset erigon stage log index since upgrading to 2.60.x
polygon-3 erigon-integration time="2024-07-09T06:26:23Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/cpu.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:26:23Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/memory.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:26:23Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/io.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:26:23Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/cpu.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:26:23Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/memory.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:26:23Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/io.pressure: operation not supported"
polygon-3 erigon-integration [INFO] [07-09|06:26:23.554] logging to file system                   log dir=/data/erigon/logs file prefix=integration log level=info json=false
polygon-3 erigon-integration time="2024-07-09T06:26:23Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/cpu.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:26:23Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/memory.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:26:23Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/io.pressure: operation not supported"
polygon-3 erigon-integration [INFO] [07-09|06:26:23.556] [db] open                                label=chaindata sizeLimit=12TB pageSize=8192
polygon-3 erigon-integration [INFO] [07-09|06:26:27.921] [snapshots:all] Stat                     blocks=56300k indices=56300k alloc=3.1GB sys=3.2GB
polygon-3 erigon-integration [INFO] [07-09|06:26:28.103] [snapshots:all] Stat                     blocks=54600k indices=54600k alloc=3.2GB sys=3.4GB
polygon-3 erigon-integration [INFO] [07-09|06:26:28.103] Opening Database                         label=bor path=/data/erigon/bor
polygon-3 erigon-integration time="2024-07-09T06:26:28Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/cpu.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:26:28Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/memory.pressure: operation not supported"
polygon-3 erigon-integration time="2024-07-09T06:26:28Z" level=error msg="unable to parse PSI data: read /sys/fs/cgroup/io.pressure: operation not supported"
polygon-3 erigon-integration [INFO] [07-09|06:26:28.107] Clear                                    table=LogAddressIndex
polygon-3 erigon-integration [INFO] [07-09|06:26:28.107] Clear                                    table=LogTopicIndex
- polygon-3 › erigon-integration

The erigon startup didn't seem to work even after running that command erigon_logs.txt

tobidae-cb avatar Jul 09 '24 14:07 tobidae-cb

@AskAlexSharov The command didn't work, it's still stuck

tobidae-cb avatar Jul 13 '24 07:07 tobidae-cb

@tobidae-cb Hi, apologies for the delay. Looking at the logs I think you've run into a known issue we've been facing recently. We have now prioritised it and are looking into fixing it. May I please get an idea of the urgency on your side - i.e. would you prefer a quick bandaid solution that may get things going now or would you be ok to wait 2-3 weeks for the fix in a future v2.60.x release? If you prefer the former then I can prep a patch branch that we speculate may solve the problem and send you a few instructions to try out.

taratorio avatar Jul 24 '24 11:07 taratorio

Hi @taratorio

Can you provide both solutions? The bandaid solution and the patch branch? I will use 2 separate pods to deploy both strategies and take a snapshot of the one that performs better. Thanks

tobidae-cb avatar Jul 24 '24 15:07 tobidae-cb

@tobidae-cb it's not about performance, it is about how long you can wait for this to be fixed - ie what is the urgency.

The bandaid solution is a patch branch & some instructions for you to try that I can give you within a day.

The proper solution that we are working on might take 2-3 weeks until it reaches a release.

taratorio avatar Jul 24 '24 15:07 taratorio

@taratorio Ah gotcha can I get the bandaid solution in that case? It's urgent as we want to start using erigon a lot more as we move away from bor

tobidae-cb avatar Jul 24 '24 15:07 tobidae-cb

@tobidae-cb ok, let's try.

Before you start would you be able to send over the full log files for the 5 nodes please (attach here if possible)? The reason I'm asking is because I think there may be 2 issues going on:

  1. getting stuck in the Bodies stage - integration stage_headers --reset seems to have helped with this
  2. getting an invalid block: mismatched receipt headers for block 48077379 error in the Execution stage after running integration stage_headers --reset

The patch branch below hopefully helps with 2), however I'd like to dig in a bit further into 1) as well.

Ok, and now let's try the patch branch on 1 of the 5 nodes. The node for which you run integration stage_headers --reset (I believe polygon-3?) seems to have gotten in a bad state. I think we can recover it with a set of commands but let's leave that for latter and pick another one. Maybe polygon-0? It is important that you haven't yet run integration stage_headers --reset on it. The steps are:

  1. stop erigon
  2. checkout and build branch e2_bor_events_fallback: git fetch -> git checkout e2_bor_events_fallback -> make erigon -> make integration, etc.
  3. if the node logs were still stuck on the sameblock_num=56299999 in the Bodies stage - then run integration stage_headers --reset
  4. start erigon

Please let us know how it goes and attach full logs here if possible after trying this out.

taratorio avatar Jul 24 '24 17:07 taratorio

@taratorio i think it’s borevent.seg file. Can try remove recent ones and “stage_bor_heimdall —reset”

AskAlexSharov avatar Jul 25 '24 01:07 AskAlexSharov

@AskAlexSharov yes, I know :) - I've ported the bor events fallback from e3 to e2 in the branch that I'm asking them to execute. It is not a recent one, it is for block ~48 million

taratorio avatar Jul 25 '24 07:07 taratorio

Ok I took a different approach so that I can at least skip all the way to block 53843475 which is the current block of the snapshot here - https://snapshots.polygon.technology/. This will at least skip syncing a couple million blocks saving some time.

I'll get feedback on the nodes in about 24hr, if it starts syncing then I'll stop here but if it also stalls similar to block ~48mil then I'll use the branch you mentioned and attempt to revive from there.

Thanks for the step by step, it's quite clear!

tobidae-cb avatar Jul 26 '24 00:07 tobidae-cb

Probably helpful logs of current progress

node-polygon git:(debug-erigon) ✗ stern 2 -c erigon | grep -v Served
+ polygon-2 › erigon
polygon-2 erigon + DATA_DIR=/data/erigon
polygon-2 erigon + EXECUTION_BOOT_NODES=enode://b8f1cc9c5d4403703fbf377116469667d2b1823c0daf16b7250aa576bacf399e42c3930ccfcb02c5df6879565a2b8931335565f0e8d3f8e72385ecf4a4bf160a@3.36.224.80:30303,enode://8729e0c825f3d9cad382555f3e46dcff21af323e89025a0e6312df541f4a9e73abfa562d64906f5e59c51fe6f0501b3e61b07979606c56329c020ed739910759@54.194.245.5:30303
polygon-2 erigon + mkdir -p /data/erigon
polygon-2 erigon + [[ false == \t\r\u\e ]]
polygon-2 erigon + echo 'Starting erigon client for polygon'
polygon-2 erigon Starting erigon client for polygon
polygon-2 erigon + exec /app/erigon --chain bor-mainnet --bor.heimdall http://localhost:1317 --rpc.txfeecap=0 --http --http.addr 0.0.0.0 '--http.vhosts=*' --http.api=engine,eth,web3,net,debug,trace,txpool,erigon,admin,bor --http.port 8546 '--http.corsdomain=*' --http.trace=true --port 30303 --datadir /data/erigon --ws --ws.port 8544 --metrics --metrics.addr 0.0.0.0 --private.api.addr 0.0.0.0:9091 --metrics.addr 0.0.0.0 --metrics.port 6062 --pprof --pprof.addr 0.0.0.0 --pprof.port 6063 --bootnodes enode://b8f1cc9c5d4403703fbf377116469667d2b1823c0daf16b7250aa576bacf399e42c3930ccfcb02c5df6879565a2b8931335565f0e8d3f8e72385ecf4a4bf160a@3.36.224.80:30303,enode://8729e0c825f3d9cad382555f3e46dcff21af323e89025a0e6312df541f4a9e73abfa562d64906f5e59c51fe6f0501b3e61b07979606c56329c020ed739910759@54.194.245.5:30303 --torrent.download.rate=120mb --torrent.staticpeers=enode://b8f1cc9c5d4403703fbf377116469667d2b1823c0daf16b7250aa576bacf399e42c3930ccfcb02c5df6879565a2b8931335565f0e8d3f8e72385ecf4a4bf160a@3.36.224.80:30303,enode://8729e0c825f3d9cad382555f3e46dcff21af323e89025a0e6312df541f4a9e73abfa562d64906f5e59c51fe6f0501b3e61b07979606c56329c020ed739910759@54.194.245.5:30303 --state.cache=16384 --rpc.batch.limit=100000 --db.read.concurrency=1536 --rpc.batch.concurrency=8 --torrent.download.slots=6 --torrent.verbosity=4 --batchSize=512M --downloader.verify --db.size.limit=12TB --db.pagesize=16kb
polygon-2 erigon [INFO] [07-25|22:38:25.130] logging to file system                   log dir=/data/erigon/logs file prefix=erigon log level=info json=false
polygon-2 erigon [INFO] [07-25|22:38:25.131] Enabling metrics export to prometheus    path=http://0.0.0.0:6062/debug/metrics/prometheus
polygon-2 erigon [INFO] [07-25|22:38:25.131] Starting pprof server                    cpu="go tool pprof -lines -http=: http://0.0.0.0:6063/debug/pprof/profile?seconds=20" heap="go tool pprof -lines -http=: http://0.0.0.0:6063/debug/pprof/heap"
polygon-2 erigon [INFO] [07-25|22:38:25.131] Build info                               git_branch=HEAD git_tag=v2.60.4 git_commit=72ab70be8b28964a8446e52b7a9975958c6726bf
polygon-2 erigon [INFO] [07-25|22:38:25.131] Starting Erigon on Bor Mainnet...
polygon-2 erigon [INFO] [07-25|22:38:25.132] Maximum peer count                       ETH=100 total=100
polygon-2 erigon [INFO] [07-25|22:38:25.132] starting HTTP APIs                       port=8546 APIs=engine,eth,web3,net,debug,trace,txpool,erigon,admin,bor
polygon-2 erigon [INFO] [07-25|22:38:25.132] torrent verbosity                        level=DBG
polygon-2 erigon [INFO] [07-25|22:38:27.235] Set global gas cap                       cap=50000000
polygon-2 erigon [INFO] [07-25|22:38:27.237] [Downloader] Running with                ipv6-enabled=true ipv4-enabled=true download.rate=120mb upload.rate=4mb
polygon-2 erigon [INFO] [07-25|22:38:27.237] Opening Database                         label=chaindata path=/data/erigon/chaindata
polygon-2 erigon [INFO] [07-25|22:38:27.240] [db] open                                label=chaindata sizeLimit=12TB pageSize=16384
polygon-2 erigon [INFO] [07-25|22:38:27.254] Initialised chain configuration          config="{ChainID: 137, Homestead: 0, DAO: <nil>, Tangerine Whistle: 0, Spurious Dragon: 0, Byzantium: 0, Constantinople: 0, Petersburg: 0, Istanbul: 3395000, Muir Glacier: 3395000, Berlin: 14750000, London: 23850000, Arrow Glacier: <nil>, Gray Glacier: <nil>, Terminal Total Difficulty: <nil>, Merge Netsplit: <nil>, Shanghai: <nil>, Cancun: <nil>, Prague: <nil>, Osaka: <nil>, Engine: bor, NoPruneContracts: map[]}" genesis=0xa9c28ce2141b56c474f1dc504bee9b01eb1bd7d1a507580d5519d4437a97de1b
polygon-2 erigon [INFO] [07-25|22:38:50.114] Initialising Ethereum protocol           network=137
polygon-2 erigon [INFO] [07-25|22:38:50.114] Opening Database                         label=bor path=/data/erigon/bor
polygon-2 erigon [INFO] [07-25|22:38:52.179] Starting private RPC server              on=0.0.0.0:9091
polygon-2 erigon [INFO] [07-25|22:38:52.179] new subscription to logs established
polygon-2 erigon [INFO] [07-25|22:38:52.181] [txpool] Started
polygon-2 erigon [INFO] [07-25|22:38:52.304] rpc filters: subscribing to Erigon events
polygon-2 erigon [INFO] [07-25|22:38:52.305] new subscription to newHeaders established
polygon-2 erigon [INFO] [07-25|22:38:52.305] New txs subscriber joined
polygon-2 erigon [EROR] [07-25|22:38:52.306] [Diagnostics] Failure in running diagnostics server err="listen tcp 0.0.0.0:6060: bind: address already in use"
polygon-2 erigon [INFO] [07-25|22:38:52.311] JsonRpc endpoint opened                  ws=true ws.compression=true grpc=false websocket.url=[::]:8544 http.url=[::]:8546
polygon-2 erigon [WARN] [07-25|22:38:52.322] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
polygon-2 erigon [WARN] [07-25|22:38:52.322] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
polygon-2 erigon [INFO] [07-25|22:38:52.332] [2/15 Headers] Waiting for headers...    from=53843475
polygon-2 erigon [INFO] [07-25|22:39:22.333] [2/15 Headers] No block headers to write in this log period number=53843475 blk/second=0.000 alloc=396.6MB sys=664.8MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:39:52.332] [2/15 Headers] Wrote block headers       number=53844053 blk/second=19.267 alloc=319.0MB sys=719.4MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:40:22.333] [2/15 Headers] Wrote block headers       number=53865090 blk/second=701.233 alloc=537.4MB sys=719.8MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:40:52.333] [2/15 Headers] Wrote block headers       number=53871652 blk/second=218.733 alloc=379.4MB sys=791.2MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:41:22.333] [2/15 Headers] Wrote block headers       number=53880337 blk/second=289.500 alloc=577.9MB sys=791.2MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:41:50.107] [p2p] GoodPeers                          eth68=1
polygon-2 erigon [INFO] [07-25|22:41:50.135] [mem] memory stats                       Rss=1.0GB Size=0B Pss=1.0GB SharedClean=2.6MB SharedDirty=0B PrivateClean=195.9MB PrivateDirty=830.0MB Referenced=1.0GB Anonymous=829.8MB Swap=0B alloc=422.5MB sys=791.2MB
polygon-2 erigon [INFO] [07-25|22:41:52.180] [txpool] stat                            pending=0 baseFee=0 queued=453 alloc=437.1MB sys=791.2MB
polygon-2 erigon [INFO] [07-25|22:41:52.333] [2/15 Headers] Wrote block headers       number=53881880 blk/second=51.433 alloc=438.6MB sys=791.2MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:42:22.333] [2/15 Headers] Wrote block headers       number=53896355 blk/second=482.500 alloc=342.3MB sys=791.2MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:42:52.333] [2/15 Headers] Wrote block headers       number=53907163 blk/second=360.267 alloc=548.5MB sys=791.2MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:43:22.333] [2/15 Headers] Wrote block headers       number=53908514 blk/second=45.033 alloc=378.6MB sys=791.5MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:43:52.333] [2/15 Headers] Wrote block headers       number=53908706 blk/second=6.400 alloc=312.8MB sys=791.5MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:44:22.333] [2/15 Headers] Wrote block headers       number=53915461 blk/second=225.167 alloc=483.8MB sys=791.5MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:44:50.107] [p2p] GoodPeers                          eth68=1
polygon-2 erigon [INFO] [07-25|22:44:50.139] [mem] memory stats                       Rss=1.1GB Size=0B Pss=1.1GB SharedClean=2.4MB SharedDirty=0B PrivateClean=241.1MB PrivateDirty=876.0MB Referenced=1.1GB Anonymous=875.9MB Swap=0B alloc=341.6MB sys=799.5MB
polygon-2 erigon [INFO] [07-25|22:44:52.180] [txpool] stat                            pending=0 baseFee=0 queued=891 alloc=358.1MB sys=799.5MB
polygon-2 erigon [INFO] [07-25|22:44:52.332] [2/15 Headers] Wrote block headers       number=53922602 blk/second=238.033 alloc=360.1MB sys=799.5MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:45:22.333] [2/15 Headers] Wrote block headers       number=53924725 blk/second=70.767 alloc=347.4MB sys=799.5MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:45:52.332] [2/15 Headers] Wrote block headers       number=53955604 blk/second=1029.300 alloc=437.5MB sys=829.3MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:46:22.333] [2/15 Headers] Wrote block headers       number=53988992 blk/second=1112.933 alloc=571.9MB sys=829.3MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:46:52.333] [2/15 Headers] Wrote block headers       number=54008291 blk/second=643.300 alloc=627.3MB sys=837.3MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:47:22.333] [2/15 Headers] Wrote block headers       number=54038784 blk/second=1016.433 alloc=348.5MB sys=854.3MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:47:50.107] [p2p] GoodPeers                          eth68=2
polygon-2 erigon [INFO] [07-25|22:47:50.144] [mem] memory stats                       Rss=1.3GB Size=0B Pss=1.3GB SharedClean=2.6MB SharedDirty=0B PrivateClean=287.8MB PrivateDirty=1021.6MB Referenced=1.3GB Anonymous=1021.6MB Swap=0B alloc=619.5MB sys=854.3MB
polygon-2 erigon [INFO] [07-25|22:47:52.180] [txpool] stat                            pending=0 baseFee=0 queued=1370 alloc=659.9MB sys=866.3MB
polygon-2 erigon [INFO] [07-25|22:47:52.332] [2/15 Headers] Wrote block headers       number=54039363 blk/second=19.300 alloc=661.5MB sys=866.3MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:48:22.333] [2/15 Headers] Wrote block headers       number=54087032 blk/second=1588.967 alloc=448.5MB sys=937.4MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:48:52.332] [2/15 Headers] Inserting headers         progress=54116142 queue=1
polygon-2 erigon [INFO] [07-25|22:49:22.336] [2/15 Headers] Inserting headers         progress=54198243 queue=1
polygon-2 erigon [INFO] [07-25|22:49:52.333] [2/15 Headers] Wrote block headers       number=54239496 blk/second=5082.133 alloc=510.5MB sys=999.7MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:50:22.333] [2/15 Headers] Wrote block headers       number=54271919 blk/second=1080.767 alloc=638.7MB sys=999.7MB invalidHeaders=0 rejectedBadHeaders=0
polygon-2 erigon [INFO] [07-25|22:50:50.107] [p2p] GoodPeers                          eth68=2
polygon-2 erigon [INFO] [07-25|22:50:50.171] [mem] memory stats                       Rss=1.7GB Size=0B Pss=1.7GB SharedClean=2.6MB SharedDirty=0B PrivateClean=383.4MB PrivateDirty=1.3GB Referenced=1.7GB Anonymous=1.3GB Swap=0B alloc=538.1MB sys=999.7MB
polygon-2 erigon [INFO] [07-25|22:50:52.180] [txpool] stat                            pending=0 baseFee=0 queued=3514 alloc=563.9MB sys=999.7MB

tobidae-cb avatar Jul 26 '24 00:07 tobidae-cb

@tobidae-cb hey, haven't heard back from you so I assume your workaround of downloading an Erigon datadir copy from the Polygon website helped and you are up and running with Erigon 2? If that is the case I'll close this issue but please do re-open it if you are not.

To give you a further update on the 2 issues you've hit here:

  • We did some investigation on issue 1) "getting stuck in block bodies stage" - a simple restart of the process fixes this (no need for integration stage_headers --reset. This error will be fixed properly in Erigon 3.
  • Regarding issue 2) "incorrect borevents data in borevents.seg file for blocks ~48 million" - we have @mh0lt working on fixing this for Erigon 3 (and maybe Erigon 2 TBD) as part of https://github.com/erigontech/erigon/issues/11188

taratorio avatar Aug 09 '24 12:08 taratorio