Geth node at some stage of synchronization sees the wrong last block

[JeySee3]

System information

Geth version: v1.10.18 OS & Version: Docker Network ropsten

Expected behaviour

Geth node must be synced and have the same actual block as on the resource.

Actual behaviour

Geth node at some stage of synchronization sees the wrong last block:

{
  currentBlock: 12307984,
  healedBytecodeBytes: "0x0",
  healedBytecodes: "0x0",
  healedTrienodeBytes: "0x0",
  healedTrienodes: "0x0",
  healingBytecode: "0x0",
  healingTrienodes: "0x0",
  highestBlock: 12363123,
  startingBlock: 0,
  syncedAccountBytes: "0x1245641c6",
  syncedAccounts: "0x1b83e256",
  syncedBytecodeBytes: "0x1b730e275",
  syncedBytecodes: "0x143984",
  syncedStorage: "0x702b11b",
  syncedStorageBytes: "0x61b629c65"
}

Steps to reproduce the behaviour

Running docker containers beacon node and geth node

version: '3.9'
services:
  beacon-node:
    image: gcr.io/prysmaticlabs/prysm/beacon-chain:v2.1.3-rc.3
    hostname: beacon-node
    restart: always
    container_name: beacon-node
    volumes:
      - "./data:/data"
      - "./genesis.ssz:/genesis/genesis.ssz"
      - "./eth-node/root/.ethereum/ropsten:/ropsten"
    command:
      - --datadir=/data
      - --rpc-host=0.0.0.0
      - --monitoring-host=0.0.0.0
      - --http-web3provider=http://eth-node:8550
      - --genesis-state=/genesis/genesis.ssz
      - --terminal-total-difficulty-override=50000000000000000
      - --ropsten
      - --accept-terms-of-use
      - --jwt-secret=/ropsten/geth/jwtsecret
    ports:
      - 4000:4000
      - 13000:13000 
      - 12000:12000/udp

  eth-node:
    image: ethereum/client-go:v1.10.18
    hostname: eth-node
    restart: always
    container_name: eth-node
    ports:
      - 30303:30303
      - 30303:30303/udp
      - 8550:8550
      - 8546:8546
      - 6060:6060
    restart: always
    volumes:
      - "./eth-node/root:/root"
    command:
      - --ropsten
      - --cache=1024
      - --nousb
      - --port=30303
      - --http
      - --http.api=admin,debug,miner,txpool,personal,eth,net,web3
      - --http.port=8550
      - --http.addr=0.0.0.0
      - --http.vhosts=*
      - --http.corsdomain=*
      - --metrics
      - --metrics.addr=0.0.0.0
      - --metrics.port=6060
      - --graphql
      - --graphql.vhosts=*
      - --ws
      - --ws.addr=0.0.0.0
      - --ws.port=8546
      - --ws.api=admin,debug,miner,txpool,personal,eth,net,web3
      - --ws.origins=*
      - --rpc.allow-unprotected-txs
      - --override.terminaltotaldifficulty=50000000000000000

Backtrace

beacon-node constantly repeating logs:

time="2022-06-17 11:39:06" level=info msg="Processing block batch of size 63 starting from  0xaaab6212... 65592/127695 - estimated time remaining 22m24s" blocksPerSecond=46.2 peers=31 prefix=initial-sync
time="2022-06-17 11:39:06" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x3cf390dad7e844346f0d12911dbd5efd007696fe8331bcd793952c9095c2840b (in processBatchedBlocks, slot=65592)" prefix=initial-sync
time="2022-06-17 11:39:06" level=error msg="Could not connect to execution client endpoint: could not make initial request to verify execution chain ID: chain not synced beyond EIP-155 replay-protection fork block" prefix=powchain
time="2022-06-17 11:39:12" level=info msg="Processing block batch of size 51 starting from  0x43636c7b... 65144/127696 - estimated time remaining 22m33s" blocksPerSecond=46.2 peers=31 prefix=initial-sync
time="2022-06-17 11:39:15" level=warning msg="Batch is not processed" error="could not process block in batch: method not found: received an undefined ee error" prefix=initial-sync
time="2022-06-17 11:39:15" level=info msg="Processing block batch of size 48 starting from  0xdf9ccf76... 65208/127696 - estimated time remaining 37m7s" blocksPerSecond=28.1 peers=31 prefix=initial-sync
time="2022-06-17 11:39:15" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x70daaf3efd184ee1c99b3af2eea39f1ee6f9b4a42a0f70fa9cc7c84181930f43 (in processBatchedBlocks, slot=65208)" prefix=initial-sync

geth-node constantly repeating logs:

INFO [06-17|08:44:04.223] Legacy sync reached merge threshold      number=12,308,106 hash=a6fbe9..bdbf28 td=43,532,321,247,089,466 ttd=43,531,756,765,713,534
INFO [06-17|08:44:04.223] State sync in progress                   synced=50.84% state=35.94GiB  accounts=465,546,[email protected]  slots=117,887,[email protected] codes=1,328,[email protected]  eta=18h41m57.108s
WARN [06-17|08:44:04.223] Local chain is post-merge, waiting for beacon client sync switch-over... 
WARN [06-17|08:44:04.250] Unexpected account range packet          peer=e8833b70 reqid=8,514,626,685,972,389,954
WARN [06-17|08:44:04.252] Unexpected account range packet          peer=139fa632 reqid=1,086,043,285,859,566,162
WARN [06-17|08:44:04.253] Unexpected storage ranges packet         peer=139fa632 reqid=7,529,167,791,326,941,287
WARN [06-17|08:44:04.253] Unexpected bytecode packet               peer=139fa632 reqid=646,791,935,132,217,832
WARN [06-17|08:44:04.256] Unexpected account range packet          peer=2c13243c reqid=7,653,476,202,871,043,935
WARN [06-17|08:44:04.263] Unexpected account range packet          peer=b0445182 reqid=4,755,942,332,859,140,285
WARN [06-17|08:44:04.288] Unexpected account range packet          peer=95281c67 reqid=7,277,932,409,281,940,899
WARN [06-17|08:44:04.325] Unexpected account range packet          peer=0729594e reqid=2,615,073,081,028,114,705
WARN [06-17|08:44:04.327] Unexpected bytecode packet               peer=86842165 reqid=8,950,867,808,705,118,033
WARN [06-17|08:44:04.419] Unexpected account range packet          peer=080fda6b reqid=2,070,293,642,309,205,639
WARN [06-17|08:44:04.420] Unexpected storage ranges packet         peer=080fda6b reqid=838,929,813,382,532,662
INFO [06-17|08:44:04.653] Disabling direct-ancient mode            origin=12,307,984 ancient=11,370,495
INFO [06-17|08:44:04.790] Imported new block headers               count=0    elapsed=9.287ms     number=12,308,105 hash=52ffbb..ee8ee8 age=3w14h15m   ignored=121

[karalabe]

Geth is just waiting for the beacon client to tell us what the head is. Your beacon client seems to be reporting it's stuck: level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x3cf390dad7e844346f0d12911dbd5efd007696fe8331bcd793952c9095c2840b (in processBatchedBlocks, slot=65592)" My guess is that your beacon node fails to sync for whatever reason?

[JeySee3]

Yes, it is, but we cannot understand why this is happening. Perhaps we somehow started the beacon node incorrectly? Or is there something else needed to make it work? Can you suggest something about this?

[basvlanat]

Hello. I'm seeing a similar problem myself. I created a task in the prism repository, but unfortunately no one answered me. @karalabe Can you help me get around it somehow?

[brainstormk]

I think that you can fix it with checkpoint switch when run beacon node by using prysm.

./prysm.sh beacon-chain --checkpoint-sync-url=http://localhost:3500 --genesis-beacon-api-url=http://localhost:3500

You can find checkpoint url on https://eth-clients.github.io/checkpoint-sync-endpoints/