avalanchego icon indicating copy to clipboard operation
avalanchego copied to clipboard

Published 20 hours ago • verified publisher icon ava-labs

bootstrap failure: block failed verification during bootstrapping due to invalid merkle root (BAD BLOCK)

Open ongrid opened this issue 2 years ago • 9 comments

Describe the bug

During node bootstrapping at C-Chain sync stage avalanchego process fails with the following output

INFO [08-11|02:57:42.055] <C Chain> github.com/ava-labs/coreth/trie/database.go:799: Persisted trie from memory database nodes=138,505 size=30.04MiB   time=1.177639876s gcnodes=1,070,119 gcsize=371.03MiB gctime=4.560253499s livenodes=11319 livesize=3.84MiB
[08-11|02:58:02.436] INFO <C Chain> queue/jobs.go:174 executed 6022500 of 18367447 operations. ETA = 29h36m47s
[08-11|02:58:36.411] INFO <C Chain> queue/jobs.go:174 executed 6025000 of 18367447 operations. ETA = 29h36m51s
INFO [08-11|02:58:40.081] <C Chain> github.com/ava-labs/coreth/plugin/evm/atomic_trie.go:411: committed atomic trie c=atomicTrie hash=0xff9c892db16b7238615071850acfd7b91ac55ad9acbe988711406ec4aea454ee height=6,025,216
INFO [08-11|02:58:40.672] <C Chain> github.com/ava-labs/coreth/trie/database.go:799: Persisted trie from memory database nodes=108,937 size=24.42MiB   time=1.072952763s gcnodes=838,790   gcsize=292.53MiB gctime=3.492957034s livenodes=8920  livesize=3.06MiB
[08-11|02:59:15.423] INFO <C Chain> queue/jobs.go:174 executed 6027500 of 18367447 operations. ETA = 29h37m5s
INFO [08-11|02:59:42.025] <C Chain> github.com/ava-labs/coreth/plugin/evm/atomic_trie.go:411: committed atomic trie c=atomicTrie hash=0x5d608dc966138806a6d64525703f73bc0bcbd43ed24de513784df260ba498259 height=6,029,312
INFO [08-11|02:59:42.565] <C Chain> github.com/ava-labs/coreth/trie/database.go:799: Persisted trie from memory database nodes=112,135 size=24.13MiB   time=1.180085588s gcnodes=822,200   gcsize=286.85MiB gctime=3.529110075s livenodes=9139  livesize=3.15MiB
[08-11|02:59:54.174] INFO <C Chain> queue/jobs.go:174 executed 6030000 of 18367447 operations. ETA = 29h37m18s
[08-11|03:00:36.308] INFO <C Chain> queue/jobs.go:174 executed 6032500 of 18367447 operations. ETA = 29h37m39s
INFO [08-11|03:00:53.364] <C Chain> github.com/ava-labs/coreth/plugin/evm/atomic_trie.go:411: committed atomic trie c=atomicTrie hash=0xfe949cf4eb60a5e05165a98f79ecf6940edb2b2733c674e211289659332b474e height=6,033,408
INFO [08-11|03:00:53.837] <C Chain> github.com/ava-labs/coreth/trie/database.go:799: Persisted trie from memory database nodes=119,471 size=26.58MiB   time=1.204507195s gcnodes=914,043   gcsize=318.75MiB gctime=4.014995093s livenodes=12765 livesize=4.28MiB
ERROR[08-11|03:00:55.439] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:1292: 
########## BAD BLOCK #########
Chain config: Chain ID:  43114
Consensus: Dummy Consensus Engine

Hard Forks:
 - Homestead:                   0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
 - DAO Fork:                    0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
 - Tangerine Whistle (EIP 150): 0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
 - Spurious Dragon/1 (EIP 155): 0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
 - Spurious Dragon/2 (EIP 158): 0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
 - Byzantium:                   0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
 - Constantinople:              0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
 - Petersburg:                  0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
 - Istanbul:                    0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
 - Muir Glacier:                0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
 - Apricot Phase 1 Timestamp:   1617199200 (https://github.com/ava-labs/avalanchego/releases/tag/v1.3.0)
 - Apricot Phase 2 Timestamp:   1620644400 (https://github.com/ava-labs/avalanchego/releases/tag/v1.4.0)
 - Apricot Phase 3 Timestamp:   1629813600 (https://github.com/ava-labs/avalanchego/releases/tag/v1.5.0)
 - Apricot Phase 4 Timestamp:   1632344400 (https://github.com/ava-labs/avalanchego/releases/tag/v1.6.0)
 - Apricot Phase 5 Timestamp:   1638468000 (https://github.com/ava-labs/avalanchego/releases/tag/v1.7.0)



Number: 6033476
Hash: 0xa5818b42b8d9a36f13c4847cdce6dac99ed87bd74e71a7b5ea1c0d53b22db2b4
         0: cumulative: 198944 gas: 198944 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x7f09e7b302d7d5e9657bf79c33effbd9c5ff49155c5f015a0ba736961d8f1bb4 logs: [0xc188017c30 0xc188017ce0 0xc188017d90 0xc188017e40 0xc188017ef0 0xc188078000 0xc1880780b0] bloom: 00000000000400000000000080000000000000000000000020000000004001000800000000000000004000000000000000800000000000000000000000200000000080000000000000000008000000000000000000000000000000008000000000000000020004000000000000000800000000000008000000000010000000000000000000000000000000000000000000000001000000080000000000000000020000000000000000000000000001000000040000000000000000000000000000000002010040000000000000020000000000000000001080000000000020000010000000020000000000000000000800000000000000600000000100000000 state: 
         1: cumulative: 399338 gas: 200394 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xd8426a75042b0857d1e55042b83c5314daee221616ba44ccd03ed04db1a2f641 logs: [0xc188078210 0xc1880782c0 0xc188078370 0xc188078420 0xc1880784d0 0xc188078580 0xc188078630 0xc1880786e0 0xc188078790 0xc188078840 0xc1880788f0] bloom: 00000000000000080000000000000000000000000400000000000000000000400a000000000002000000002000004100088000000000000000000000080000000000000000000000000008084000000200000000000000000000000001000000000000000200000000000000000008000000000000000000002010100002000000000000000000000000000000000008000000000000000000000000000000040000000000040200000000000000010000000400000000000000010000040000000082020000000000000000000000000001000000000040800000000000200000000010010a0004000000001000000100000000400000000000000000000000 state: 


Error: invalid merkle root (remote: 34cbd19559cacacf5a29037612eef63d3181dd73ce8b20d866e912e736c5ec5d local: 0f1158117b4ca9dbc3b8df08f8c0bc385c5ccdafa90dfd20b409de84232cc89a)
##############################
 
[08-11|03:00:55.439] ERROR <C Chain> bootstrap/block_job.go:86 block 2c5ViUspnWLeW1ugZyuBhyAQeidP39x8vKK7Uycyb5THaCtBJH failed verification during bootstrapping due to invalid merkle root (remote: 34cbd19559cacacf5a29037612eef63d3181dd73ce8b20d866e912e736c5ec5d local: 0f1158117b4ca9dbc3b8df08f8c0bc385c5ccdafa90dfd20b409de84232cc89a)
[08-11|03:00:55.444] FATAL <C Chain> handler/handler.go:276 shutting down chain due to unexpected error: failed to execute job 2c5ViUspnWLeW1ugZyuBhyAQeidP39x8vKK7Uycyb5THaCtBJH due to failed to verify block in bootstrapping: invalid merkle root (remote: 34cbd19559cacacf5a29037612eef63d3181dd73ce8b20d866e912e736c5ec5d local: 0f1158117b4ca9dbc3b8df08f8c0bc385c5ccdafa90dfd20b409de84232cc89a) while processing sync message: (Op: ancestors, NodeID: NodeID-8nZEi6MTsNRJeikac1brUsQo74DgeZMed, RequestID: 567486, NumContainers: 9)
[08-11|03:00:55.444] INFO <C Chain> bootstrap/bootstrapper.go:258 shutting down bootstrapper
INFO [08-11|03:00:55.448] <C Chain> github.com/ava-labs/coreth/core/tx_pool.go:457: Transaction pool stopped 
INFO [08-11|03:00:55.448] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:688: Stopping Acceptor 
INFO [08-11|03:00:55.448] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:691: Acceptor queue drained t="2.289µs"
INFO [08-11|03:00:55.449] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:693: Shutting down state manager 
INFO [08-11|03:00:55.500] <C Chain> github.com/ava-labs/coreth/trie/database.go:799: Persisted trie from memory database nodes=6726    size=1.98MiB    time=50.947388ms  gcnodes=13912     gcsize=4.80MiB   gctime=47.908276ms  livenodes=5547  livesize=1.97MiB
INFO [08-11|03:00:55.500] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:698: State manager shut down t=51.064663ms
INFO [08-11|03:00:55.500] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:701: Shutting down sender cacher 
INFO [08-11|03:00:55.501] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:705: Closing scope 
INFO [08-11|03:00:55.501] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:708: Blockchain stopped 
[08-11|03:00:55.502] INFO node/node.go:1287 shutting down node with exit code 1
[08-11|03:00:55.505] INFO ipcs/chainipc.go:101 shutting down chain IPCs
[08-11|03:00:55.508] INFO chains/manager.go:988 shutting down chain manager
[08-11|03:00:55.508] INFO router/chain_router.go:265 shutting down chain router
[08-11|03:00:55.508] INFO <P Chain> bootstrap/bootstrapper.go:258 shutting down bootstrapper
[08-11|03:00:55.508] INFO <X Chain> bootstrap/bootstrapper.go:257 shutting down bootstrapper
[08-11|03:00:55.511] INFO network/network.go:1028 shutting down the p2p networking
[08-11|03:00:55.562] INFO node/node.go:1330 cleaning up plugin subprocesses
[08-11|03:00:55.578] INFO node/node.go:1340 finished node shutdown
[08-11|03:00:55.579] INFO nat/nat.go:152 Unmapped all ports

To Reproduce build avalanchego from 0950acab667e0c16a55e9a9bb72bcbe25c3b88cf (master, v1.7.16 candidate)

./scripts/build.sh

then run

avalanchego --public-ip=<ip_address>

Expected behavior

Node expected to pass bootstrapping process and transition to healthy state.

Logs Node start:

[08-08|06:46:51.018] INFO node/node.go:1177 node version is: avalanche/1.7.16
[08-08|06:46:51.018] INFO node/node.go:1178 node ID is: NodeID-GfuVf7kpevtMWjNm4vuXmi7CdbNn2dzBV
[08-08|06:46:51.018] INFO node/node.go:552 initializing API server
[08-08|06:46:51.019] INFO server/server.go:136 API created with allowed origins: [*]
[08-08|06:46:51.019] INFO node/node.go:849 initializing metrics API
[08-08|06:46:51.019] INFO server/server.go:271 adding route /ext/metrics
[08-08|06:46:51.019] INFO leveldb/db.go:199 leveldb config: {"blockCacheCapacity":12582912,"blockSize":0,"compactionExpandLimitFactor":0,"compactionGPOverlapsFactor":0,"compactionL0Trigger":0,"compactionSourceLimitFactor":0,"compactionTableSize":0,"compactionTableSizeMultiplier":0,"compactionTableSizeMultiplierPerLevel":null,"compactionTotalSize":0,"compactionTotalSizeMultiplier":0,"disableSeeksCompaction":true,"openFilesCacheCapacity":1024,"writeBuffer":6291456,"filterBitsPerKey":10,"maxManifestFileSize":9223372036854775807,"metricUpdateFrequency":10000000000}
[08-08|06:46:51.029] INFO node/node.go:446 current database version: v1.4.5
[08-08|06:46:51.042] INFO node/node.go:803 initializing keystore
[08-08|06:46:51.042] INFO node/node.go:814 initializing keystore API
[08-08|06:46:51.042] INFO server/server.go:271 adding route /ext/keystore
[08-08|06:46:51.043] INFO node/node.go:795 initializing SharedMemory
[08-08|06:46:51.044] INFO node/node.go:210 this node's IP is set to: "3.236.96.156:9651"
[08-08|06:46:51.045] INFO node/node.go:967 initializing Health API
[08-08|06:46:51.045] INFO server/server.go:271 adding route /ext/health
[08-08|06:46:51.045] INFO server/server.go:271 adding route /ext/health/readiness
[08-08|06:46:51.045] INFO server/server.go:271 adding route /ext/health/health
[08-08|06:46:51.045] INFO server/server.go:271 adding route /ext/health/liveness
[08-08|06:46:51.045] INFO node/node.go:599 adding the default VM aliases
[08-08|06:46:51.330] INFO node/node.go:716 initializing VMs
[08-08|06:46:51.330] INFO server/server.go:271 adding route /ext/vm/rWhpuQPF1kb72esV2momhMuTYGkEb1oL29pt2EBXWmSy4kxnT
[08-08|06:46:51.330] INFO server/server.go:271 adding route /ext/vm/jvYyfQTxGMJLuGWa55kdP2p2zSUYsQ5Raupu4TW34ZAUBAbtq
[08-08|06:46:51.330] INFO server/server.go:271 adding route /ext/vm/mgj786NP7uDwBCcq6YwThhaN8FLyybkCa4zBWTQbNgmK6k9A6/rpc
[08-08|06:46:51.331] INFO node/node.go:873 node config:
{"httpConfig":{"apiConfig":{"authConfig":{"apiRequireAuthToken":false},"indexerConfig":{"indexAPIEnabled":false,"indexAllowIncomplete":false},"ipcConfig":{"ipcAPIEnabled":false,"ipcPath":"/tmp","ipcDefaultChainIDs":null},"adminAPIEnabled":false,"infoAPIEnabled":true,"keystoreAPIEnabled":true,"metricsAPIEnabled":true,"healthAPIEnabled":true},"httpHost":"127.0.0.1","httpPort":9650,"httpsEnabled":false,"apiAllowedOrigins":["*"],"shutdownTimeout":10000000000,"shutdownWait":0},"ipConfig":{"ip":{"ip":"3.236.96.156","port":9651},"ipResolutionFrequency":300000000000,"attemptedNATTraversal":false},"stakingConfig":{"uptimeRequirement":0.8,"minValidatorStake":2000000000000,"maxValidatorStake":3000000000000000,"minDelegatorStake":25000000000,"minDelegationFee":20000,"minStakeDuration":1209600000000000,"maxStakeDuration":31536000000000000,"rewardConfig":{"maxConsumptionRate":120000,"minConsumptionRate":100000,"mintingPeriod":31536000000000000,"supplyCap":720000000000000000},"enableStaking":true,"disabledStakingWeight":100,"stakingKeyPath":"/root/.avalanchego/staking/staker.key","stakingCertPath":"/root/.avalanchego/staking/staker.crt"},"txFeeConfig":{"txFee":1000000,"createAssetTxFee":10000000,"createSubnetTxFee":1000000000,"createBlockchainTxFee":1000000000},"stateSyncConfig":{"stateSyncIDs":null,"stateSyncIPs":null},"bootstrapConfig":{"retryBootstrap":true,"retryBootstrapWarnFrequency":50,"bootstrapBeaconConnectionTimeout":60000000000,"bootstrapAncestorsMaxContainersSent":2000,"bootstrapAncestorsMaxContainersReceived":2000,"bootstrapMaxTimeGetAncestors":50000000,"bootstrapIDs":["NodeID-HiFv1DpKXkAAfJ1NHWVqQoojjznibZXHP","NodeID-A6onFGyJjA37EZ7kYHANMR1PFRT8NmXrF","NodeID-9CkG9MBNavnw7EVSRsuFr7ws9gascDQy3","NodeID-GSgaA47umS1px2ohVjodW9621Ks63xDxD","NodeID-FYv1Lb29SqMpywYXH7yNkcFAzRF2jvm3K"],"bootstrapIPs":[{"ip":"15.207.11.193","port":9651},{"ip":"54.94.43.49","port":9651},{"ip":"3.128.138.36","port":9651},{"ip":"18.229.206.191","port":9651},{"ip":"15.188.9.42","port":9651}]},"databaseConfig":{"path":"/root/.avalanchego/db/mainnet","name":"leveldb"},"avaxAssetID":"FvwEAhmxKfeiG8SnEvq42hc6whRyY3EFYAvebMqDNDGCgxN5Z","networkID":1,"enableAssertions":true,"enableCrypto":true,"healthCheckFreq":30000000000,"networkConfig":{"healthConfig":{"minConnectedPeers":1,"maxTimeSinceMsgReceived":60000000000,"maxTimeSinceMsgSent":60000000000,"maxPortionSendQueueBytesFull":0.9,"maxSendFailRate":0.9,"sendFailRateHalflife":10000000000},"peerListGossipConfig":{"peerListNumValidatorIPs":15,"peerListValidatorGossipSize":20,"peerListNonValidatorGossipSize":10,"peerListPeersGossipSize":0,"peerListGossipFreq":60000000000},"timeoutConfigs":{"pingPongTimeout":30000000000,"readHandshakeTimeout":15000000000},"delayConfig":{"initialReconnectDelay":1000000000,"maxReconnectDelay":3600000000000},"throttlerConfig":{"inboundConnUpgradeThrottlerConfig":{"upgradeCooldown":10000000000,"maxRecentConnsUpgraded":2560},"inboundMsgThrottlerConfig":{"byteThrottlerConfig":{"vdrAllocSize":33554432,"atLargeAllocSize":6291456,"nodeMaxAtLargeBytes":2097152},"bandwidthThrottlerConfig":{"bandwidthRefillRate":524288,"bandwidthMaxBurstRate":2097152},"cpuThrottlerConfig":{"maxRecheckDelay":5000000000},"diskThrottlerConfig":{"maxRecheckDelay":5000000000},"maxProcessingMsgsPerNode":1024},"outboundMsgThrottlerConfig":{"vdrAllocSize":33554432,"atLargeAllocSize":33554432,"nodeMaxAtLargeBytes":2097152},"maxInboundConnsPerSec":256},"dialerConfig":{"throttleRps":50,"connectionTimeout":30000000000},"namespace":"network","myNodeID":"NodeID-GfuVf7kpevtMWjNm4vuXmi7CdbNn2dzBV","myIP":{"ip":"3.236.96.156","port":9651},"networkID":1,"maxClockDifference":60000000000,"pingFrequency":22500000000,"allowPrivateIPs":true,"compressionEnabled":true,"whitelistedSubnets":[],"beacons":{},"validators":{},"uptimeMetricFreq":30000000000,"uptimeRequirement":0.8,"requireValidatorToConnect":false,"maximumInboundMessageTimeout":10000000000,"peerReadBufferSize":8192,"peerWriteBufferSize":8192},"gossipConfig":{"gossipAcceptedFrontierValidatorSize":0,"gossipAcceptedFrontierNonValidatorSize":0,"gossipAcceptedFrontierPeerSize":15,"gossipOnAcceptValidatorSize":0,"gossipOnAcceptNonValidatorSize":0,"gossipOnAcceptPeerSize":10,"appGossipValidatorSize":10,"appGossipNonValidatorSize":0,"appGossipPeerSize":0},"adaptiveTimeoutConfig":{"initialTimeout":5000000000,"minimumTimeout":2000000000,"maximumTimeout":10000000000,"timeoutCoefficient":2,"timeoutHalflife":300000000000},"benchlistConfig":{"threshold":10,"minimumFailingDuration":150000000000,"duration":900000000000,"maxPortion":0.08333333333333333},"profilerConfig":{"dir":"/root/.avalanchego/profiles","enabled":false,"freq":900000000000,"maxNumFiles":5},"loggingConfig":{"maxSize":8,"maxFiles":7,"maxAge":0,"directory":"/root/.avalanchego/logs","compress":false,"disableWriterDisplaying":false,"assertions":false,"logLevel":"INFO","displayLevel":"INFO","logFormat":"PLAIN"},"pluginDir":"/usr/local/lib/avalanchego/plugins","fdLimit":32768,"consensusParams":{"k":20,"alpha":15,"betaVirtuous":15,"betaRogue":20,"concurrentRepolls":4,"optimalProcessing":50,"maxOutstandingItems":1024,"maxItemProcessingTime":120000000000,"mixedQueryNumPushVdr":10,"mixedQueryNumPushNonVdr":0,"parents":5,"batchSize":30},"meterVMEnabled":true,"routerHealthConfig":{"maxDropRate":1,"maxDropRateHalflife":10000000000,"maxOutstandingRequests":1024,"maxOutstandingDuration":300000000000,"maxRunTimeRequests":10000000000},"consensusShutdownTimeout":30000000000,"consensusGossipFreq":10000000000,"whitelistedSubnets":[],"subnetConfigs":{},"systemTrackerProcessingHalflife":15000000000,"systemTrackerFrequency":500000000,"systemTrackerCPUHalflife":15000000000,"systemTrackerDiskHalflife":60000000000,"cpuTargeterConfig":{"vdrAlloc":8,"maxNonVdrUsage":6.4,"maxNonVdrNodeUsage":1},"diskTargeterConfig":{"vdrAlloc":1073741824000,"maxNonVdrUsage":1073741824000,"maxNonVdrNodeUsage":1073741824000},"requiredAvailableDiskSpace":536870912,"warningThresholdAvailableDiskSpace":1073741824}
[08-08|06:46:51.331] INFO node/node.go:875 skipping admin API initialization because it has been disabled
[08-08|06:46:51.331] INFO node/node.go:925 initializing info API
[08-08|06:46:51.331] INFO server/server.go:271 adding route /ext/info
[08-08|06:46:51.331] INFO node/node.go:1068 skipping ipc API initialization because it has been disabled
[08-08|06:46:51.331] INFO node/node.go:1081 initializing chain aliases
[08-08|06:46:51.476] INFO node/node.go:1099 initializing API aliases
[08-08|06:46:51.625] INFO node/node.go:900 skipping profiler initialization because it has been disabled
[08-08|06:46:51.625] INFO node/node.go:538 initializing chains
[08-08|06:46:51.625] INFO chains/manager.go:262 creating chain:
    ID: 11111111111111111111111111111111LpoYY
    VMID:rWhpuQPF1kb72esV2momhMuTYGkEb1oL29pt2EBXWmSy4kxnT
[08-08|06:46:52.283] INFO <P Chain> platformvm/vm.go:176 initializing last accepted block as 2FUFPVPxbTpKNn39moGSzsmGroYES4NZRdw3mJgNvMkMiMHJ9e
[08-08|06:46:52.283] INFO <P Chain> snowman/transitive.go:69 initializing consensus engine
[08-08|06:46:52.284] INFO server/server.go:248 adding route /ext/bc/11111111111111111111111111111111LpoYY
[08-08|06:46:52.284] INFO <P Chain> bootstrap/bootstrapper.go:112 Starting bootstrap...
[08-08|06:46:52.284] INFO server/server.go:167 HTTP API server listening on "127.0.0.1:9650"
[08-08|06:46:57.966] INFO <P Chain> common/bootstrapper.go:224 Bootstrapping started syncing with 1 vertices in the accepted frontier
[08-08|06:47:03.306] INFO <P Chain> bootstrap/bootstrapper.go:442 fetched 5000 of 1987691 blocks. ETA = 35m17s
[08-08|06:47:08.956] INFO <P Chain> bootstrap/bootstrapper.go:442 fetched 10000 of 1987691 blocks. ETA = 36m13s
[08-08|06:47:15.816] INFO <P Chain> bootstrap/bootstrapper.go:442 fetched 15000 of 1987691 blocks. ETA = 39m7s
[08-08|06:47:21.758] INFO <P Chain> bootstrap/bootstrapper.go:442 fetched 20000 of 1987691 blocks. ETA = 39m1s
[08-08|06:47:28.113] INFO <P Chain> bootstrap/bootstrapper.go:442 fetched 25000 of 1987691 blocks. ETA = 39m27s

last lines:

[08-11|02:58:02.436] INFO <C Chain> queue/jobs.go:174 executed 6022500 of 18367447 operations. ETA = 29h36m47s
[08-11|02:58:36.411] INFO <C Chain> queue/jobs.go:174 executed 6025000 of 18367447 operations. ETA = 29h36m51s
INFO [08-11|02:58:40.081] <C Chain> github.com/ava-labs/coreth/plugin/evm/atomic_trie.go:411: committed atomic trie c=atomicTrie hash=0xff9c892db16b7238615071850acfd7b91ac55ad9acbe988711406ec4aea454ee height=6,025,216
INFO [08-11|02:58:40.672] <C Chain> github.com/ava-labs/coreth/trie/database.go:799: Persisted trie from memory database nodes=108,937 size=24.42MiB   time=1.072952763s gcnodes=838,790   gcsize=292.53MiB gctime=3.492957034s livenodes=8920  livesize=3.06MiB
[08-11|02:59:15.423] INFO <C Chain> queue/jobs.go:174 executed 6027500 of 18367447 operations. ETA = 29h37m5s
INFO [08-11|02:59:42.025] <C Chain> github.com/ava-labs/coreth/plugin/evm/atomic_trie.go:411: committed atomic trie c=atomicTrie hash=0x5d608dc966138806a6d64525703f73bc0bcbd43ed24de513784df260ba498259 height=6,029,312
INFO [08-11|02:59:42.565] <C Chain> github.com/ava-labs/coreth/trie/database.go:799: Persisted trie from memory database nodes=112,135 size=24.13MiB   time=1.180085588s gcnodes=822,200   gcsize=286.85MiB gctime=3.529110075s livenodes=9139  livesize=3.15MiB
[08-11|02:59:54.174] INFO <C Chain> queue/jobs.go:174 executed 6030000 of 18367447 operations. ETA = 29h37m18s
[08-11|03:00:36.308] INFO <C Chain> queue/jobs.go:174 executed 6032500 of 18367447 operations. ETA = 29h37m39s
INFO [08-11|03:00:53.364] <C Chain> github.com/ava-labs/coreth/plugin/evm/atomic_trie.go:411: committed atomic trie c=atomicTrie hash=0xfe949cf4eb60a5e05165a98f79ecf6940edb2b2733c674e211289659332b474e height=6,033,408
INFO [08-11|03:00:53.837] <C Chain> github.com/ava-labs/coreth/trie/database.go:799: Persisted trie from memory database nodes=119,471 size=26.58MiB   time=1.204507195s gcnodes=914,043   gcsize=318.75MiB gctime=4.014995093s livenodes=12765 livesize=4.28MiB
ERROR[08-11|03:00:55.439] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:1292: 
########## BAD BLOCK #########
Chain config: Chain ID:  43114
Consensus: Dummy Consensus Engine

Hard Forks:
 - Homestead:                   0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
 - DAO Fork:                    0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
 - Tangerine Whistle (EIP 150): 0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
 - Spurious Dragon/1 (EIP 155): 0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
 - Spurious Dragon/2 (EIP 158): 0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
 - Byzantium:                   0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
 - Constantinople:              0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
 - Petersburg:                  0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
 - Istanbul:                    0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
 - Muir Glacier:                0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
 - Apricot Phase 1 Timestamp:   1617199200 (https://github.com/ava-labs/avalanchego/releases/tag/v1.3.0)
 - Apricot Phase 2 Timestamp:   1620644400 (https://github.com/ava-labs/avalanchego/releases/tag/v1.4.0)
 - Apricot Phase 3 Timestamp:   1629813600 (https://github.com/ava-labs/avalanchego/releases/tag/v1.5.0)
 - Apricot Phase 4 Timestamp:   1632344400 (https://github.com/ava-labs/avalanchego/releases/tag/v1.6.0)
 - Apricot Phase 5 Timestamp:   1638468000 (https://github.com/ava-labs/avalanchego/releases/tag/v1.7.0)



Number: 6033476
Hash: 0xa5818b42b8d9a36f13c4847cdce6dac99ed87bd74e71a7b5ea1c0d53b22db2b4
         0: cumulative: 198944 gas: 198944 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x7f09e7b302d7d5e9657bf79c33effbd9c5ff49155c5f015a0ba736961d8f1bb4 logs: [0xc188017c30 0xc188017ce0 0xc188017d90 0xc188017e40 0xc188017ef0 0xc188078000 0xc1880780b0] bloom: 00000000000400000000000080000000000000000000000020000000004001000800000000000000004000000000000000800000000000000000000000200000000080000000000000000008000000000000000000000000000000008000000000000000020004000000000000000800000000000008000000000010000000000000000000000000000000000000000000000001000000080000000000000000020000000000000000000000000001000000040000000000000000000000000000000002010040000000000000020000000000000000001080000000000020000010000000020000000000000000000800000000000000600000000100000000 state: 
         1: cumulative: 399338 gas: 200394 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xd8426a75042b0857d1e55042b83c5314daee221616ba44ccd03ed04db1a2f641 logs: [0xc188078210 0xc1880782c0 0xc188078370 0xc188078420 0xc1880784d0 0xc188078580 0xc188078630 0xc1880786e0 0xc188078790 0xc188078840 0xc1880788f0] bloom: 00000000000000080000000000000000000000000400000000000000000000400a000000000002000000002000004100088000000000000000000000080000000000000000000000000008084000000200000000000000000000000001000000000000000200000000000000000008000000000000000000002010100002000000000000000000000000000000000008000000000000000000000000000000040000000000040200000000000000010000000400000000000000010000040000000082020000000000000000000000000001000000000040800000000000200000000010010a0004000000001000000100000000400000000000000000000000 state: 


Error: invalid merkle root (remote: 34cbd19559cacacf5a29037612eef63d3181dd73ce8b20d866e912e736c5ec5d local: 0f1158117b4ca9dbc3b8df08f8c0bc385c5ccdafa90dfd20b409de84232cc89a)
##############################
 
[08-11|03:00:55.439] ERROR <C Chain> bootstrap/block_job.go:86 block 2c5ViUspnWLeW1ugZyuBhyAQeidP39x8vKK7Uycyb5THaCtBJH failed verification during bootstrapping due to invalid merkle root (remote: 34cbd19559cacacf5a29037612eef63d3181dd73ce8b20d866e912e736c5ec5d local: 0f1158117b4ca9dbc3b8df08f8c0bc385c5ccdafa90dfd20b409de84232cc89a)
[08-11|03:00:55.444] FATAL <C Chain> handler/handler.go:276 shutting down chain due to unexpected error: failed to execute job 2c5ViUspnWLeW1ugZyuBhyAQeidP39x8vKK7Uycyb5THaCtBJH due to failed to verify block in bootstrapping: invalid merkle root (remote: 34cbd19559cacacf5a29037612eef63d3181dd73ce8b20d866e912e736c5ec5d local: 0f1158117b4ca9dbc3b8df08f8c0bc385c5ccdafa90dfd20b409de84232cc89a) while processing sync message: (Op: ancestors, NodeID: NodeID-8nZEi6MTsNRJeikac1brUsQo74DgeZMed, RequestID: 567486, NumContainers: 9)
[08-11|03:00:55.444] INFO <C Chain> bootstrap/bootstrapper.go:258 shutting down bootstrapper
INFO [08-11|03:00:55.448] <C Chain> github.com/ava-labs/coreth/core/tx_pool.go:457: Transaction pool stopped 
INFO [08-11|03:00:55.448] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:688: Stopping Acceptor 
INFO [08-11|03:00:55.448] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:691: Acceptor queue drained t="2.289µs"
INFO [08-11|03:00:55.449] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:693: Shutting down state manager 
INFO [08-11|03:00:55.500] <C Chain> github.com/ava-labs/coreth/trie/database.go:799: Persisted trie from memory database nodes=6726    size=1.98MiB    time=50.947388ms  gcnodes=13912     gcsize=4.80MiB   gctime=47.908276ms  livenodes=5547  livesize=1.97MiB
INFO [08-11|03:00:55.500] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:698: State manager shut down t=51.064663ms
INFO [08-11|03:00:55.500] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:701: Shutting down sender cacher 
INFO [08-11|03:00:55.501] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:705: Closing scope 
INFO [08-11|03:00:55.501] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:708: Blockchain stopped 
[08-11|03:00:55.502] INFO node/node.go:1287 shutting down node with exit code 1
[08-11|03:00:55.505] INFO ipcs/chainipc.go:101 shutting down chain IPCs
[08-11|03:00:55.508] INFO chains/manager.go:988 shutting down chain manager
[08-11|03:00:55.508] INFO router/chain_router.go:265 shutting down chain router
[08-11|03:00:55.508] INFO <P Chain> bootstrap/bootstrapper.go:258 shutting down bootstrapper
[08-11|03:00:55.508] INFO <X Chain> bootstrap/bootstrapper.go:257 shutting down bootstrapper
[08-11|03:00:55.511] INFO network/network.go:1028 shutting down the p2p networking
[08-11|03:00:55.562] INFO node/node.go:1330 cleaning up plugin subprocesses
[08-11|03:00:55.578] INFO node/node.go:1340 finished node shutdown
[08-11|03:00:55.579] INFO nat/nat.go:152 Unmapped all ports

Operating System

built on golang:1.17.9-buster (Docker) executed on debian:11-slim (Docker)

Additional context

Restart of avalanchego leads to shutdown by the same reason

[08-11|08:06:49.672] INFO node/node.go:1177 node version is: avalanche/1.7.16
[08-11|08:06:49.672] INFO node/node.go:1178 node ID is: NodeID-GfuVf7kpevtMWjNm4vuXmi7CdbNn2dzBV
[08-11|08:06:49.672] INFO node/node.go:552 initializing API server
[08-11|08:06:49.672] INFO server/server.go:136 API created with allowed origins: [*]
[08-11|08:06:49.672] INFO node/node.go:849 initializing metrics API
[08-11|08:06:49.673] INFO server/server.go:271 adding route /ext/metrics
[08-11|08:06:49.673] INFO leveldb/db.go:199 leveldb config: {"blockCacheCapacity":12582912,"blockSize":0,"compactionExpandLimitFactor":0,"compactionGPOverlapsFactor":0,"compactionL0Trigger":0,"compactionSourceLimitFactor":0,"compactionTableSize":0,"compactionTableSizeMultiplier":0,"compactionTableSizeMultiplierPerLevel":null,"compactionTotalSize":0,"compactionTotalSizeMultiplier":0,"disableSeeksCompaction":true,"openFilesCacheCapacity":1024,"writeBuffer":6291456,"filterBitsPerKey":10,"maxManifestFileSize":9223372036854775807,"metricUpdateFrequency":10000000000}
[08-11|08:06:50.228] INFO node/node.go:446 current database version: v1.4.5
[08-11|08:06:50.246] INFO node/node.go:803 initializing keystore
[08-11|08:06:50.246] INFO node/node.go:814 initializing keystore API
[08-11|08:06:50.246] INFO server/server.go:271 adding route /ext/keystore
[08-11|08:06:50.247] INFO node/node.go:795 initializing SharedMemory
[08-11|08:06:50.250] INFO node/node.go:210 this node's IP is set to: "3.236.96.156:9651"
[08-11|08:06:50.252] INFO node/node.go:967 initializing Health API
[08-11|08:06:50.253] INFO server/server.go:271 adding route /ext/health
[08-11|08:06:50.253] INFO server/server.go:271 adding route /ext/health/readiness
[08-11|08:06:50.253] INFO server/server.go:271 adding route /ext/health/health
[08-11|08:06:50.253] INFO server/server.go:271 adding route /ext/health/liveness
[08-11|08:06:50.253] INFO node/node.go:599 adding the default VM aliases
[08-11|08:06:50.670] INFO node/node.go:716 initializing VMs
[08-11|08:06:50.670] INFO server/server.go:271 adding route /ext/vm/rWhpuQPF1kb72esV2momhMuTYGkEb1oL29pt2EBXWmSy4kxnT
[08-11|08:06:50.670] INFO server/server.go:271 adding route /ext/vm/jvYyfQTxGMJLuGWa55kdP2p2zSUYsQ5Raupu4TW34ZAUBAbtq
[08-11|08:06:50.670] INFO server/server.go:271 adding route /ext/vm/mgj786NP7uDwBCcq6YwThhaN8FLyybkCa4zBWTQbNgmK6k9A6/rpc
[08-11|08:06:50.671] INFO node/node.go:873 node config:
{"httpConfig":{"apiConfig":{"authConfig":{"apiRequireAuthToken":false},"indexerConfig":{"indexAPIEnabled":false,"indexAllowIncomplete":false},"ipcConfig":{"ipcAPIEnabled":false,"ipcPath":"/tmp","ipcDefaultChainIDs":null},"adminAPIEnabled":false,"infoAPIEnabled":true,"keystoreAPIEnabled":true,"metricsAPIEnabled":true,"healthAPIEnabled":true},"httpHost":"127.0.0.1","httpPort":9650,"httpsEnabled":false,"apiAllowedOrigins":["*"],"shutdownTimeout":10000000000,"shutdownWait":0},"ipConfig":{"ip":{"ip":"3.236.96.156","port":9651},"ipResolutionFrequency":300000000000,"attemptedNATTraversal":false},"stakingConfig":{"uptimeRequirement":0.8,"minValidatorStake":2000000000000,"maxValidatorStake":3000000000000000,"minDelegatorStake":25000000000,"minDelegationFee":20000,"minStakeDuration":1209600000000000,"maxStakeDuration":31536000000000000,"rewardConfig":{"maxConsumptionRate":120000,"minConsumptionRate":100000,"mintingPeriod":31536000000000000,"supplyCap":720000000000000000},"enableStaking":true,"disabledStakingWeight":100,"stakingKeyPath":"/root/.avalanchego/staking/staker.key","stakingCertPath":"/root/.avalanchego/staking/staker.crt"},"txFeeConfig":{"txFee":1000000,"createAssetTxFee":10000000,"createSubnetTxFee":1000000000,"createBlockchainTxFee":1000000000},"stateSyncConfig":{"stateSyncIDs":null,"stateSyncIPs":null},"bootstrapConfig":{"retryBootstrap":true,"retryBootstrapWarnFrequency":50,"bootstrapBeaconConnectionTimeout":60000000000,"bootstrapAncestorsMaxContainersSent":2000,"bootstrapAncestorsMaxContainersReceived":2000,"bootstrapMaxTimeGetAncestors":50000000,"bootstrapIDs":["NodeID-9CkG9MBNavnw7EVSRsuFr7ws9gascDQy3","NodeID-HsBEx3L71EHWSXaE6gvk2VsNntFEZsxqc","NodeID-4kCLS16Wy73nt1Zm54jFZsL7Msrv3UCeJ","NodeID-AaxT2P4uuPAHb7vAD8mNvjQ3jgyaV7tu9","NodeID-Nr584bLpGgbCUbZFSBaBz3Xum5wpca9Ym"],"bootstrapIPs":[{"ip":"3.128.138.36","port":9651},{"ip":"3.106.25.139","port":9651},{"ip":"122.248.199.127","port":9651},{"ip":"54.185.87.50","port":9651},{"ip":"18.162.129.129","port":9651}]},"databaseConfig":{"path":"/root/.avalanchego/db/mainnet","name":"leveldb"},"avaxAssetID":"FvwEAhmxKfeiG8SnEvq42hc6whRyY3EFYAvebMqDNDGCgxN5Z","networkID":1,"enableAssertions":true,"enableCrypto":true,"healthCheckFreq":30000000000,"networkConfig":{"healthConfig":{"minConnectedPeers":1,"maxTimeSinceMsgReceived":60000000000,"maxTimeSinceMsgSent":60000000000,"maxPortionSendQueueBytesFull":0.9,"maxSendFailRate":0.9,"sendFailRateHalflife":10000000000},"peerListGossipConfig":{"peerListNumValidatorIPs":15,"peerListValidatorGossipSize":20,"peerListNonValidatorGossipSize":10,"peerListPeersGossipSize":0,"peerListGossipFreq":60000000000},"timeoutConfigs":{"pingPongTimeout":30000000000,"readHandshakeTimeout":15000000000},"delayConfig":{"initialReconnectDelay":1000000000,"maxReconnectDelay":3600000000000},"throttlerConfig":{"inboundConnUpgradeThrottlerConfig":{"upgradeCooldown":10000000000,"maxRecentConnsUpgraded":2560},"inboundMsgThrottlerConfig":{"byteThrottlerConfig":{"vdrAllocSize":33554432,"atLargeAllocSize":6291456,"nodeMaxAtLargeBytes":2097152},"bandwidthThrottlerConfig":{"bandwidthRefillRate":524288,"bandwidthMaxBurstRate":2097152},"cpuThrottlerConfig":{"maxRecheckDelay":5000000000},"diskThrottlerConfig":{"maxRecheckDelay":5000000000},"maxProcessingMsgsPerNode":1024},"outboundMsgThrottlerConfig":{"vdrAllocSize":33554432,"atLargeAllocSize":33554432,"nodeMaxAtLargeBytes":2097152},"maxInboundConnsPerSec":256},"dialerConfig":{"throttleRps":50,"connectionTimeout":30000000000},"namespace":"network","myNodeID":"NodeID-GfuVf7kpevtMWjNm4vuXmi7CdbNn2dzBV","myIP":{"ip":"3.236.96.156","port":9651},"networkID":1,"maxClockDifference":60000000000,"pingFrequency":22500000000,"allowPrivateIPs":true,"compressionEnabled":true,"whitelistedSubnets":[],"beacons":{},"validators":{},"uptimeMetricFreq":30000000000,"uptimeRequirement":0.8,"requireValidatorToConnect":false,"maximumInboundMessageTimeout":10000000000,"peerReadBufferSize":8192,"peerWriteBufferSize":8192},"gossipConfig":{"gossipAcceptedFrontierValidatorSize":0,"gossipAcceptedFrontierNonValidatorSize":0,"gossipAcceptedFrontierPeerSize":15,"gossipOnAcceptValidatorSize":0,"gossipOnAcceptNonValidatorSize":0,"gossipOnAcceptPeerSize":10,"appGossipValidatorSize":10,"appGossipNonValidatorSize":0,"appGossipPeerSize":0},"adaptiveTimeoutConfig":{"initialTimeout":5000000000,"minimumTimeout":2000000000,"maximumTimeout":10000000000,"timeoutCoefficient":2,"timeoutHalflife":300000000000},"benchlistConfig":{"threshold":10,"minimumFailingDuration":150000000000,"duration":900000000000,"maxPortion":0.08333333333333333},"profilerConfig":{"dir":"/root/.avalanchego/profiles","enabled":false,"freq":900000000000,"maxNumFiles":5},"loggingConfig":{"maxSize":8,"maxFiles":7,"maxAge":0,"directory":"/root/.avalanchego/logs","compress":false,"disableWriterDisplaying":false,"assertions":false,"logLevel":"INFO","displayLevel":"INFO","logFormat":"PLAIN"},"pluginDir":"/usr/local/lib/avalanchego/plugins","fdLimit":32768,"consensusParams":{"k":20,"alpha":15,"betaVirtuous":15,"betaRogue":20,"concurrentRepolls":4,"optimalProcessing":50,"maxOutstandingItems":1024,"maxItemProcessingTime":120000000000,"mixedQueryNumPushVdr":10,"mixedQueryNumPushNonVdr":0,"parents":5,"batchSize":30},"meterVMEnabled":true,"routerHealthConfig":{"maxDropRate":1,"maxDropRateHalflife":10000000000,"maxOutstandingRequests":1024,"maxOutstandingDuration":300000000000,"maxRunTimeRequests":10000000000},"consensusShutdownTimeout":30000000000,"consensusGossipFreq":10000000000,"whitelistedSubnets":[],"subnetConfigs":{},"systemTrackerProcessingHalflife":15000000000,"systemTrackerFrequency":500000000,"systemTrackerCPUHalflife":15000000000,"systemTrackerDiskHalflife":60000000000,"cpuTargeterConfig":{"vdrAlloc":8,"maxNonVdrUsage":6.4,"maxNonVdrNodeUsage":1},"diskTargeterConfig":{"vdrAlloc":1073741824000,"maxNonVdrUsage":1073741824000,"maxNonVdrNodeUsage":1073741824000},"requiredAvailableDiskSpace":536870912,"warningThresholdAvailableDiskSpace":1073741824}
[08-11|08:06:50.671] INFO node/node.go:875 skipping admin API initialization because it has been disabled
[08-11|08:06:50.671] INFO node/node.go:925 initializing info API
[08-11|08:06:50.671] INFO server/server.go:271 adding route /ext/info
[08-11|08:06:50.671] INFO node/node.go:1068 skipping ipc API initialization because it has been disabled
[08-11|08:06:50.671] INFO node/node.go:1081 initializing chain aliases
[08-11|08:06:50.810] INFO node/node.go:1099 initializing API aliases
[08-11|08:06:50.977] INFO node/node.go:900 skipping profiler initialization because it has been disabled
[08-11|08:06:50.977] INFO node/node.go:538 initializing chains
[08-11|08:06:50.977] INFO chains/manager.go:262 creating chain:
    ID: 11111111111111111111111111111111LpoYY
    VMID:rWhpuQPF1kb72esV2momhMuTYGkEb1oL29pt2EBXWmSy4kxnT
[08-11|08:06:55.158] INFO <P Chain> platformvm/vm.go:176 initializing last accepted block as unZnfTzazejvuTgXJ8X9vJ8CJWL7WQujAXBapGvj7yGNGGHcJ
[08-11|08:06:55.186] INFO <P Chain> snowman/transitive.go:69 initializing consensus engine
[08-11|08:06:55.187] INFO server/server.go:248 adding route /ext/bc/11111111111111111111111111111111LpoYY
[08-11|08:06:55.187] INFO <P Chain> bootstrap/bootstrapper.go:112 Starting bootstrap...
[08-11|08:06:55.187] INFO server/server.go:167 HTTP API server listening on "127.0.0.1:9650"
[08-11|08:06:55.243] INFO peer/peer.go:639 beacon NodeID-9CkG9MBNavnw7EVSRsuFr7ws9gascDQy3 attempting to connect with newer version avalanche/1.7.17. You may want to update your client
[08-11|08:06:55.408] INFO peer/peer.go:639 beacon NodeID-AaxT2P4uuPAHb7vAD8mNvjQ3jgyaV7tu9 attempting to connect with newer version avalanche/1.7.17. You may want to update your client
[08-11|08:06:55.813] INFO peer/peer.go:639 beacon NodeID-HsBEx3L71EHWSXaE6gvk2VsNntFEZsxqc attempting to connect with newer version avalanche/1.7.17. You may want to update your client
[08-11|08:06:55.820] INFO peer/peer.go:639 beacon NodeID-Nr584bLpGgbCUbZFSBaBz3Xum5wpca9Ym attempting to connect with newer version avalanche/1.7.17. You may want to update your client
[08-11|08:06:55.852] INFO peer/peer.go:639 beacon NodeID-4kCLS16Wy73nt1Zm54jFZsL7Msrv3UCeJ attempting to connect with newer version avalanche/1.7.17. You may want to update your client
[08-11|08:07:01.227] INFO <P Chain> common/bootstrapper.go:224 Bootstrapping started syncing with 1 vertices in the accepted frontier
[08-11|08:07:01.820] INFO <P Chain> bootstrap/bootstrapper.go:493 bootstrapping fetched 65 blocks. Executing state transitions...
[08-11|08:07:01.865] INFO <P Chain> queue/jobs.go:182 executed 65 operations
[08-11|08:07:02.289] INFO chains/manager.go:262 creating chain:
    ID: 2q9e4r6Mu3U68nU1fYjgbR6JvwrRx36CohpAX5UQxse55x1Q5
    VMID:mgj786NP7uDwBCcq6YwThhaN8FLyybkCa4zBWTQbNgmK6k9A6
INFO [08-11|08:07:02.291] <C Chain> github.com/ava-labs/coreth/plugin/evm/vm.go:386: Initializing Coreth VM Version=v0.8.14 Config="{\"snowman-api-enabled\":false,\"coreth-admin-api-enabled\":false,\"coreth-admin-api-dir\":\"\",\"eth-apis\":[\"eth\",\"eth-filter\",\"net\",\"web3\",\"internal-eth\",\"internal-blockchain\",\"internal-transaction\"],\"continuous-profiler-dir\":\"\",\"continuous-profiler-frequency\":{\"Duration\":900000000000},\"continuous-profiler-max-files\":5,\"rpc-gas-cap\":50000000,\"rpc-tx-fee-cap\":100,\"preimages-enabled\":false,\"snapshot-async\":true,\"snapshot-verification-enabled\":false,\"pruning-enabled\":true,\"accepted-queue-limit\":64,\"commit-interval\":4096,\"allow-missing-tries\":false,\"populate-missing-tries-parallelism\":1024,\"metrics-expensive-enabled\":false,\"local-txs-enabled\":false,\"api-max-duration\":{\"Duration\":0},\"ws-cpu-refill-rate\":{\"Duration\":0},\"ws-cpu-max-stored\":{\"Duration\":0},\"api-max-blocks-per-request\":0,\"allow-unfinalized-queries\":false,\"allow-unprotected-txs\":false,\"keystore-directory\":\"\",\"keystore-external-signer\":\"\",\"keystore-insecure-unlock-allowed\":false,\"remote-tx-gossip-only-enabled\":false,\"tx-regossip-frequency\":{\"Duration\":60000000000},\"tx-regossip-max-size\":15,\"log-level\":\"info\",\"offline-pruning-enabled\":false,\"offline-pruning-bloom-filter-size\":512,\"offline-pruning-data-directory\":\"\",\"max-outbound-active-requests\":16,\"state-sync-enabled\":false,\"state-sync-skip-resume\":false,\"state-sync-server-trie-cache\":64,\"state-sync-ids\":\"\",\"state-sync-commit-interval\":16384,\"state-sync-min-blocks\":300000}"
INFO [08-11|08:07:02.293] <C Chain> github.com/ava-labs/coreth/trie/database.go:799: Persisted trie from memory database nodes=1 size=140.00B time="10.897µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [08-11|08:07:02.293] <C Chain> github.com/ava-labs/coreth/plugin/evm/vm.go:491: lastAccepted = 0x0e897ae464da4e177fbd6d36e444aa2624744182900794c8d2c8285e96b8f0d3 
INFO [08-11|08:07:02.293] <C Chain> github.com/ava-labs/coreth/eth/backend.go:149: Allocated trie memory caches clean=256.00MiB dirty=256.00MiB
INFO [08-11|08:07:02.294] <C Chain> github.com/ava-labs/coreth/trie/database.go:799: Persisted trie from memory database nodes=1 size=140.00B time="7.39µs"   gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:159:  
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:160: --------------------------------------------------------------------------------------------------------------------------------------------------------- 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162: Chain ID:  43114 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162: Consensus: Dummy Consensus Engine 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162:  
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162: Hard Forks: 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162:  - Homestead:                   0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md) 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162:  - DAO Fork:                    0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md) 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162:  - Tangerine Whistle (EIP 150): 0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md) 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162:  - Spurious Dragon/1 (EIP 155): 0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md) 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162:  - Spurious Dragon/2 (EIP 158): 0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md) 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162:  - Byzantium:                   0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md) 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162:  - Constantinople:              0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md) 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162:  - Petersburg:                  0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md) 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162:  - Istanbul:                    0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md) 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162:  - Muir Glacier:                0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md) 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162:  - Apricot Phase 1 Timestamp:   1617199200 (https://github.com/ava-labs/avalanchego/releases/tag/v1.3.0) 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162:  - Apricot Phase 2 Timestamp:   1620644400 (https://github.com/ava-labs/avalanchego/releases/tag/v1.4.0) 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162:  - Apricot Phase 3 Timestamp:   1629813600 (https://github.com/ava-labs/avalanchego/releases/tag/v1.5.0) 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162:  - Apricot Phase 4 Timestamp:   1632344400 (https://github.com/ava-labs/avalanchego/releases/tag/v1.6.0) 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162:  - Apricot Phase 5 Timestamp:   1638468000 (https://github.com/ava-labs/avalanchego/releases/tag/v1.7.0) 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162:  
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:162:  
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:164: --------------------------------------------------------------------------------------------------------------------------------------------------------- 
INFO [08-11|08:07:02.295] <C Chain> github.com/ava-labs/coreth/eth/backend.go:165:  
INFO [08-11|08:07:02.296] <C Chain> github.com/ava-labs/coreth/eth/backend.go:196: Initialising Ethereum protocol network=43114 dbversion=8
INFO [08-11|08:07:02.297] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:492: Loaded most recent local header number=6,033,475 hash=0e897a..b8f0d3 age=9mo3w1h
INFO [08-11|08:07:02.297] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:493: Loaded most recent local full block number=6,033,475 hash=0e897a..b8f0d3 age=9mo3w1h
INFO [08-11|08:07:02.297] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:1414: Loaded Acceptor tip hash=0e897a..b8f0d3
INFO [08-11|08:07:02.297] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:1422: Skipping state reprocessing root=9fd836..959c68
INFO [08-11|08:07:02.297] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:1396: Initializing snapshots async=true rebuild=true headHash=0e897a..b8f0d3 headRoot=9fd836..959c68
INFO [08-11|08:07:02.298] <C Chain> github.com/ava-labs/coreth/core/tx_pool.go:504: Transaction pool price threshold updated price=470,000,000,000
INFO [08-11|08:07:02.298] <C Chain> github.com/ava-labs/coreth/core/tx_pool.go:504: Transaction pool price threshold updated price=225,000,000,000
INFO [08-11|08:07:02.298] <C Chain> github.com/ava-labs/coreth/core/tx_pool.go:504: Transaction pool price threshold updated price=0
INFO [08-11|08:07:02.298] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:359: Starting Acceptor queue length=64
INFO [08-11|08:07:02.299] <C Chain> github.com/ava-labs/coreth/plugin/evm/atomic_trie.go:233: initializing atomic trie c=atomicTrie lastAcceptedBlockNumber=6,033,475
INFO [08-11|08:07:02.299] <C Chain> github.com/ava-labs/coreth/plugin/evm/atomic_trie.go:351: finished initializing atomic trie c=atomicTrie lastAcceptedBlockNumber=6,033,475 preCommitEntriesIndexed=0 postCommitEntriesIndexed=3 lastCommittedHash=fe949c..2b474e lastCommittedHeight=6,033,408 time="356.273µs"
[08-11|08:07:02.300] INFO <C Chain> proposervm/vm.go:304 block height index was successfully verified
[08-11|08:07:02.302] INFO <C Chain> snowman/transitive.go:69 initializing consensus engine
INFO [08-11|08:07:02.303] <C Chain> github.com/ava-labs/coreth/plugin/evm/vm.go:1194: Enabled APIs: eth, eth-filter, net, web3, internal-eth, internal-blockchain, internal-transaction, avax 
[08-11|08:07:02.303] INFO server/server.go:248 adding route /ext/bc/2q9e4r6Mu3U68nU1fYjgbR6JvwrRx36CohpAX5UQxse55x1Q5/avax
[08-11|08:07:02.303] INFO server/server.go:248 adding route /ext/bc/2q9e4r6Mu3U68nU1fYjgbR6JvwrRx36CohpAX5UQxse55x1Q5/rpc
[08-11|08:07:02.303] INFO server/server.go:248 adding route /ext/bc/2q9e4r6Mu3U68nU1fYjgbR6JvwrRx36CohpAX5UQxse55x1Q5/ws
[08-11|08:07:02.303] INFO <C Chain> bootstrap/bootstrapper.go:112 Starting bootstrap...
[08-11|08:07:02.304] INFO chains/manager.go:262 creating chain:
    ID: 2oYMBNV4eNHyqk2fjjV5nVQLDbtmNJzq5s3qs3Lo6ftnC6FByM
    VMID:jvYyfQTxGMJLuGWa55kdP2p2zSUYsQ5Raupu4TW34ZAUBAbtq
[08-11|08:07:02.324] INFO <X Chain> avm/vm.go:454 Fee payments are using Asset with Alias: AVAX, AssetID: FvwEAhmxKfeiG8SnEvq42hc6whRyY3EFYAvebMqDNDGCgxN5Z
[08-11|08:07:02.324] INFO <X Chain> avm/vm.go:236 address transaction indexing is disabled
[08-11|08:07:02.325] INFO <X Chain> avalanche/transitive.go:73 initializing consensus engine
[08-11|08:07:02.326] INFO server/server.go:248 adding route /ext/bc/2oYMBNV4eNHyqk2fjjV5nVQLDbtmNJzq5s3qs3Lo6ftnC6FByM
[08-11|08:07:02.326] INFO server/server.go:248 adding route /ext/bc/2oYMBNV4eNHyqk2fjjV5nVQLDbtmNJzq5s3qs3Lo6ftnC6FByM/wallet
[08-11|08:07:02.326] INFO server/server.go:248 adding route /ext/bc/2oYMBNV4eNHyqk2fjjV5nVQLDbtmNJzq5s3qs3Lo6ftnC6FByM/events
[08-11|08:07:02.326] INFO <X Chain> bootstrap/bootstrapper.go:264 Starting bootstrap...
[08-11|08:07:24.542] INFO <X Chain> common/bootstrapper.go:224 Bootstrapping started syncing with 1 vertices in the accepted frontier
[08-11|08:07:24.642] INFO <C Chain> common/bootstrapper.go:224 Bootstrapping started syncing with 1 vertices in the accepted frontier
[08-11|08:07:25.756] INFO <X Chain> bootstrap/bootstrapper.go:477 bootstrapping fetched 54 vertices. Executing transaction state transitions...
[08-11|08:07:25.812] INFO <X Chain> queue/jobs.go:182 executed 54 operations
[08-11|08:07:25.813] INFO <X Chain> bootstrap/bootstrapper.go:488 executing vertex state transitions...
[08-11|08:07:25.821] INFO <X Chain> queue/jobs.go:182 executed 54 operations
[08-11|08:07:34.065] INFO <C Chain> bootstrap/bootstrapper.go:493 bootstrapping fetched 12462445 blocks. Executing state transitions...
ERROR[08-11|08:07:34.087] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:1292: 
########## BAD BLOCK #########
Chain config: Chain ID:  43114
Consensus: Dummy Consensus Engine

Hard Forks:
 - Homestead:                   0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
 - DAO Fork:                    0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
 - Tangerine Whistle (EIP 150): 0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
 - Spurious Dragon/1 (EIP 155): 0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
 - Spurious Dragon/2 (EIP 158): 0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
 - Byzantium:                   0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
 - Constantinople:              0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
 - Petersburg:                  0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
 - Istanbul:                    0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
 - Muir Glacier:                0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
 - Apricot Phase 1 Timestamp:   1617199200 (https://github.com/ava-labs/avalanchego/releases/tag/v1.3.0)
 - Apricot Phase 2 Timestamp:   1620644400 (https://github.com/ava-labs/avalanchego/releases/tag/v1.4.0)
 - Apricot Phase 3 Timestamp:   1629813600 (https://github.com/ava-labs/avalanchego/releases/tag/v1.5.0)
 - Apricot Phase 4 Timestamp:   1632344400 (https://github.com/ava-labs/avalanchego/releases/tag/v1.6.0)
 - Apricot Phase 5 Timestamp:   1638468000 (https://github.com/ava-labs/avalanchego/releases/tag/v1.7.0)



Number: 6033476
Hash: 0xa5818b42b8d9a36f13c4847cdce6dac99ed87bd74e71a7b5ea1c0d53b22db2b4
         0: cumulative: 198944 gas: 198944 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x7f09e7b302d7d5e9657bf79c33effbd9c5ff49155c5f015a0ba736961d8f1bb4 logs: [0xc010a6de40 0xc010a6def0 0xc011f92000 0xc011f920b0 0xc011f92160 0xc011f92210 0xc011f922c0] bloom: 00000000000400000000000080000000000000000000000020000000004001000800000000000000004000000000000000800000000000000000000000200000000080000000000000000008000000000000000000000000000000008000000000000000020004000000000000000800000000000008000000000010000000000000000000000000000000000000000000000001000000080000000000000000020000000000000000000000000001000000040000000000000000000000000000000002010040000000000000020000000000000000001080000000000020000010000000020000000000000000000800000000000000600000000100000000 state: 
         1: cumulative: 399338 gas: 200394 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xd8426a75042b0857d1e55042b83c5314daee221616ba44ccd03ed04db1a2f641 logs: [0xc011f92420 0xc011f924d0 0xc011f92580 0xc011f92630 0xc011f926e0 0xc011f92790 0xc011f92840 0xc00bcab1e0 0xc00bcab290 0xc00bcab340 0xc00bcab3f0] bloom: 00000000000000080000000000000000000000000400000000000000000000400a000000000002000000002000004100088000000000000000000000080000000000000000000000000008084000000200000000000000000000000001000000000000000200000000000000000008000000000000000000002010100002000000000000000000000000000000000008000000000000000000000000000000040000000000040200000000000000010000000400000000000000010000040000000082020000000000000000000000000001000000000040800000000000200000000010010a0004000000001000000100000000400000000000000000000000 state: 


Error: invalid merkle root (remote: 34cbd19559cacacf5a29037612eef63d3181dd73ce8b20d866e912e736c5ec5d local: 0f1158117b4ca9dbc3b8df08f8c0bc385c5ccdafa90dfd20b409de84232cc89a)
##############################
 
[08-11|08:07:34.087] ERROR <C Chain> bootstrap/block_job.go:86 block 2c5ViUspnWLeW1ugZyuBhyAQeidP39x8vKK7Uycyb5THaCtBJH failed verification during bootstrapping due to invalid merkle root (remote: 34cbd19559cacacf5a29037612eef63d3181dd73ce8b20d866e912e736c5ec5d local: 0f1158117b4ca9dbc3b8df08f8c0bc385c5ccdafa90dfd20b409de84232cc89a)
[08-11|08:07:34.087] FATAL <C Chain> handler/handler.go:276 shutting down chain due to unexpected error: failed to execute job 2c5ViUspnWLeW1ugZyuBhyAQeidP39x8vKK7Uycyb5THaCtBJH due to failed to verify block in bootstrapping: invalid merkle root (remote: 34cbd19559cacacf5a29037612eef63d3181dd73ce8b20d866e912e736c5ec5d local: 0f1158117b4ca9dbc3b8df08f8c0bc385c5ccdafa90dfd20b409de84232cc89a) while processing sync message: (Op: ancestors, NodeID: NodeID-Hhkq3c3UzzX3UpwC2kq9TcbJdguht164A, RequestID: 11, NumContainers: 408)
[08-11|08:07:34.087] INFO <C Chain> bootstrap/bootstrapper.go:258 shutting down bootstrapper
INFO [08-11|08:07:34.087] <C Chain> github.com/ava-labs/coreth/core/tx_pool.go:457: Transaction pool stopped 
INFO [08-11|08:07:34.087] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:688: Stopping Acceptor 
INFO [08-11|08:07:34.087] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:691: Acceptor queue drained t="1.531µs"
INFO [08-11|08:07:34.087] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:693: Shutting down state manager 
INFO [08-11|08:07:34.087] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:698: State manager shut down t="1.16µs"
INFO [08-11|08:07:34.087] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:701: Shutting down sender cacher 
INFO [08-11|08:07:34.087] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:705: Closing scope 
INFO [08-11|08:07:34.087] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:708: Blockchain stopped 
[08-11|08:07:34.087] INFO node/node.go:1287 shutting down node with exit code 1
[08-11|08:07:34.087] INFO ipcs/chainipc.go:101 shutting down chain IPCs
[08-11|08:07:34.088] INFO chains/manager.go:988 shutting down chain manager
[08-11|08:07:34.088] INFO router/chain_router.go:265 shutting down chain router
[08-11|08:07:34.088] INFO <P Chain> bootstrap/bootstrapper.go:258 shutting down bootstrapper
[08-11|08:07:34.088] INFO <X Chain> bootstrap/bootstrapper.go:257 shutting down bootstrapper
[08-11|08:07:34.088] INFO network/network.go:1028 shutting down the p2p networking
[08-11|08:07:34.123] INFO node/node.go:1330 cleaning up plugin subprocesses
[08-11|08:07:34.144] INFO node/node.go:1340 finished node shutdown
[08-11|08:07:34.144] INFO nat/nat.go:152 Unmapped all ports

Similar issues:

#989, #1096, #1078, #1083

ongrid avatar Aug 11 '22 10:08 ongrid

[08-11|08:06:49.672] INFO node/node.go:1177 node version is: avalanche/1.7.16

Please make sure that you are on the latest version of 1.7.17.

yulin-dong avatar Aug 11 '22 13:08 yulin-dong

@ongrid do you have a database backup anywhere? If you share the db, we can repro!

patrick-ogrady avatar Aug 15 '22 23:08 patrick-ogrady

same issue, I am using v1.7.17

[08-16|05:26:59.835] INFO <C Chain> bootstrap/bootstrapper.go:469 fetching blocks {"numFetchedBlocks": 13050000, "numTotalBlocks": 13057538, "eta": "35s"} ^@[08-16|05:27:25.894] INFO <C Chain> bootstrap/bootstrapper.go:469 fetching blocks {"numFetchedBlocks": 13055000, "numTotalBlocks": 13057538, "eta": "12s"} [08-16|05:27:47.223] INFO <C Chain> bootstrap/bootstrapper.go:528 executing blocks {"numPendingJobs": 13057538} ERROR[08-16|05:27:47.254] <C Chain> github.com/ava-labs/coreth/core/blockchain.go:1292: ########## BAD BLOCK ######### Chain config: Chain ID: 43114 Consensus: Dummy Consensus Engine

Hard Forks:

  • Homestead: 0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
  • DAO Fork: 0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
  • Tangerine Whistle (EIP 150): 0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
  • Spurious Dragon/1 (EIP 155): 0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
  • Spurious Dragon/2 (EIP 158): 0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
  • Byzantium: 0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
  • Constantinople: 0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
  • Petersburg: 0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
  • Istanbul: 0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
  • Muir Glacier: 0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
  • Apricot Phase 1 Timestamp: 1617199200 (https://github.com/ava-labs/avalanchego/releases/tag/v1.3.0)
  • Apricot Phase 2 Timestamp: 1620644400 (https://github.com/ava-labs/avalanchego/releases/tag/v1.4.0)
  • Apricot Phase 3 Timestamp: 1629813600 (https://github.com/ava-labs/avalanchego/releases/tag/v1.5.0)
  • Apricot Phase 4 Timestamp: 1632344400 (https://github.com/ava-labs/avalanchego/releases/tag/v1.6.0)
  • Apricot Phase 5 Timestamp: 1638468000 (https://github.com/ava-labs/avalanchego/releases/tag/v1.7.0)

Number: 5644272 Hash: 0xc133cdb7d51559871473546c1dd4050d83ae5fded185db0537e50b9f989ff11e 0: cumulative: 164488 gas: 164488 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xf4dc782718d185e030fa4f731740afac54b193180c1e9c22e7899c915df81618 logs: [0xc00657cbb0 0xc00657cc60 0xc00657cd10 0xc00657cdc0 0xc00657ce70 0xc00657cf20 0xc00657cfd0] bloom: 00200000000000000000000080000000000000000000000020000080000000000000000000000000004000000000000000000000000080000000000000010000000080000000000000000008000000200000000000000000000000100000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000080000044000001000200000000000000000000000004000000000000010001000000000000100000000000002000000040002000000000010000000000000001000000000000000000020010000000000000004000000000024000000000000000000000400000800 state: 1: cumulative: 196489 gas: 32001 contract: 0x0000000000000000000000000000000000000000 status: 0 tx: 0x507fb985185130ce70b454ee891afeb22aba36969616901a48ef275abbc64d0f logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 2: cumulative: 237263 gas: 40774 contract: 0x0000000000000000000000000000000000000000 status: 0 tx: 0xfdbfb5b65b50c7bf785a0fe8c44aedef8823a967a4e55235ec3c63ebf1fd3d5d logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 3: cumulative: 400443 gas: 163180 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x4fe68e348267701c2c5ff36ff42d6f8fac87c129c2eeb35675e6ca22bdcf1292 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 4: cumulative: 461106 gas: 60663 contract: 0x0000000000000000000000000000000000000000 status: 0 tx: 0x5e848a0b4dc9a2a13a52876982ad3b688644621164669cd3493b7b063b55cf55 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 5: cumulative: 495491 gas: 34385 contract: 0x0000000000000000000000000000000000000000 status: 0 tx: 0x42c1a30320abe960e5b9650d3ee093c4eb7806d7b330f4025d4fad6e416e3c5e logs: [] bloom: 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

kmalloc avatar Aug 16 '22 04:08 kmalloc

is there a way to SetHead() like ethereum? so that I can rewind a few block?

kmalloc avatar Aug 16 '22 04:08 kmalloc

@kmalloc do you have the database that is exhibiting this issue? It would help us greatly if you do!

patrick-ogrady avatar Aug 16 '22 04:08 patrick-ogrady

Any idea how I can send you the database?it is 94GB in size.

kmalloc avatar Aug 16 '22 06:08 kmalloc

Any idea how I can send you the database?it is 94GB in size. You can zip and share on a cloud drive if you can.

yulin-dong avatar Aug 16 '22 13:08 yulin-dong

unfortunately this doesn't seem feasible for me, you can easily reproduce the issue, just start syncing from scratch. it will manifest itself after a day or two.

kmalloc avatar Aug 17 '22 07:08 kmalloc

We were able to repro. We are looking into it. Stay tuned.

patrick-ogrady avatar Aug 18 '22 18:08 patrick-ogrady

This is resolved in v1.7.18.

StephenButtolph avatar Aug 19 '22 09:08 StephenButtolph