Checkpoint sync eventually fails

[pablomendezroyo]

💎 Issue

Background

Context and background information on the discussion...

Running Prysm with https://github.com/dappnode/DAppNodePackage-prysm-prater/blob/master/beacon-chain/entrypoint.sh

Description

The checkpoint sync feature works most of the time, however, it may fail eventually becoming into an exit loop not allowing the beacon chain process to start syncing.

There are 2 main things to discuss here:

• Should the process start syncing from genesis if the checkpoint sync fails instead of keeping exiting?
• why did it fail? The error logs:

ime="2022-06-06 12:04:04" level=info msg="Updating DB and creating buckets..." prefix=db
time="2022-06-06 12:04:04" level=info msg="Updated db and created buckets" elapsed=14.022554ms prefix=db
time="2022-06-06 12:04:04" level=info msg="requesting https://[email protected]/eth/v2/debug/beacon/states/genesis"
time="2022-06-06 12:04:09" level=info msg="requesting https://[email protected]/eth/v1/beacon/weak_subjectivity"
time="2022-06-06 12:04:09" level=info msg="falling back to generic checkpoint derivation, weak_subjectivity API not supported by server"
time="2022-06-06 12:04:09" level=info msg="requesting https://[email protected]/eth/v1/node/version"
time="2022-06-06 12:04:09" level=info msg="requesting https://[email protected]/eth/v2/debug/beacon/states/head"
time="2022-06-06 12:04:12" level=info msg="detected supported config in remote head state, name=prater, fork=altair"
time="2022-06-06 12:04:12" level=info msg="(computed client-side) weak subjectivity epoch = 98875"
time="2022-06-06 12:04:12" level=info msg="requesting checkpoint state at slot 3164000"
time="2022-06-06 12:04:12" level=info msg="requesting https://[email protected]/eth/v2/debug/beacon/states/3164000"
time="2022-06-06 12:04:15" level=info msg="detected supported config in checkpoint state, name=prater, fork=altair"
time="2022-06-06 12:04:17" level=info msg="requesting https://[email protected]/eth/v2/beacon/blocks/0x95a5ffa7e33a60ef57e14a3dc2573e7aca2ec65e00c8c1dc7bc3fe3e7079eed6"
time="2022-06-06 12:04:17" level=info msg="BeaconState slot=3164000, Block slot=3164000"
time="2022-06-06 12:04:17" level=info msg="BeaconState htr=0x686965537a5bbe46d1a279c3abd132058babc5bbadbe800cd1ebd36c56afe6cad, Block state_root=0x686965537a5bbe46d1a279c3abd132058babc5bbadbe800cd1ebd36c56afe6ca"
time="2022-06-06 12:04:17" level=info msg="BeaconBlock root computed from state=0x95a5ffa7e33a60ef57e14a3dc2573e7aca2ec65e00c8c1dc7bc3fe3e7079eed6, Block htr=0x95a5ffa7e33a60ef57e14a3dc2573e7aca2ec65e00c8c1dc7bc3fe3e7079eed6"
time="2022-06-06 12:04:17" level=info msg="detected supported config for state & block version, config name=prater, fork name=altair" prefix=db
time="2022-06-06 12:04:17" level=info msg="saving checkpoint block to db, w/ root=0x95a5ffa7e33a60ef57e14a3dc2573e7aca2ec65e00c8c1dc7bc3fe3e7079eed6" prefix=db
time="2022-06-06 12:04:17" level=info msg="calling SaveState w/ blockRoot=95a5ffa7e33a60ef57e14a3dc2573e7aca2ec65e00c8c1dc7bc3fe3e7079eed6" prefix=db
time="2022-06-06 12:04:18" level=error msg="could not mark checkpoint sync block as finalized: could not retrieve block roots: invalid end slot and start slot provided" prefix=main

[kasey]

Thanks for reporting this issue @pablomendezroyo . Could you please specify the version of prysm the shell script is starting, as well as the remote implementation name and version? You can get the former by running beacon-chain --version and the latter by running curl $CHECKPOINT_SYNC_URL/eth/v1/node/version

edit: I see from the logs that this is an infura node, so ignore the second half of my comment.

[pablomendezroyo]

Running v2.1.2 https://github.com/dappnode/DAppNodePackage-prysm-prater/blob/d344676ed79832139ab20ce936512b404853818f/docker-compose.yml#L8

[nisdas]

@pablomendezroyo Is this still an issue in v2.1.3 ?

[pablomendezroyo]

Is working fine in version v3.1.2.

Thanks for the great work!