lighthouse Deposit Cache Finalization & Fast WS Sync

Summary

The deposit cache now has the ability to finalize deposits. This will cause it to drop unneeded deposit logs and hashes in the deposit Merkle tree that are no longer required to construct deposit proofs. The cache is finalized whenever the latest finalized checkpoint has a new Eth1Data with all deposits imported.

This has three benefits:

Improves the speed of constructing Merkle proofs for deposits as we can just replay deposits since the last finalized checkpoint instead of all historical deposits when re-constructing the Merkle tree.
Significantly faster weak subjectivity sync as the deposit cache can be transferred to the newly syncing node in compressed form. The Merkle tree that stores N finalized deposits requires a maximum of log2(N) hashes. The newly syncing node then only needs to download deposits since the last finalized checkpoint to have a full tree.
Future proofing in preparation for EIP-4444 as execution nodes will no longer be required to store logs permanently so we won't always have all historical logs available to us.

More Details

Image to illustrate how the deposit contract merkle tree evolves and finalizes along with the resulting DepositTreeSnapshot

Other Considerations

I've changed the structure of the SszDepositCache so once you load & save your database from this version of lighthouse, you will no longer be able to load it from older versions.

Jan 14 '22 00:01 ethDreamer

Looks like a few of the tests aren't compiling currently

Jan 18 '22 01:01 michaelsproul

Looking really good!

There are only two things I'd like to resolve before we merge:

What to do about the database schema change, i.e. this comment. I think you mentioned something in Amsterdam about why this wasn't required, but I didn't follow the reasoning. Do you think we should take the --eth1-purge-cache approach for downgrades? I still prefer the explicit schema migration, which would be v9 -> v10, with a downgrade v10 -> v9. The lighthouse db migrate command is in unstable now, so we'd just need to add the upgrade and downgrade to schema_change.rs. And we'll still have the option of removing them later as in Mac's recent PR #3181.

Whether we can impl Encode/Decode for Option<T> (this comment). I'll ping Paul to bring his attention to it.

The other thing to discuss is release timing. I think we could either ship this in v2.3.0 along with the v9 schema upgrade for #3157 so that most user's only perceive a single upgrade v8 -> v10, or we wait until after the merge. Currently there's no urgent push for v2.3.0, so we may be able to get this in before.

We actually don't need to impl Encode/Decode for Option<T> anymore because I restructured the object so they don't have Option<T> - I can delete this but we could also commit it for use later..

May 27 '22 19:05 ethDreamer

I'm going to drop this back to work-in-progress due to some conflicts. Feel free to change the tags as you see fit!

Jul 04 '22 00:07 paulhauner

I've moved the schema migration from v12 to v13 so this PR can't be merged until

#3312

is merged and we rebase with unstable. But this should be good to go for review anyway.

Aug 16 '22 20:08 ethDreamer

Finally submitted a PR to beacon-apis for the endpoint:

https://github.com/ethereum/beacon-APIs/pull/245

Sep 20 '22 20:09 ethDreamer

bors r+

Oct 29 '22 22:10 michaelsproul

Build failed:

check-msrv

Oct 29 '22 23:10 bors[bot]

dang, when you have a sec @ethDreamer looks like there's some fixups required after the latest refactor

Oct 29 '22 23:10 michaelsproul

clippy unhappy as well

Oct 29 '22 23:10 michaelsproul

@michaelsproul looks like another spurious failure of the windows tests.

Oct 30 '22 03:10 ethDreamer

let's try bors!

bors r+

Oct 30 '22 04:10 michaelsproul

Pull request successfully merged into unstable.

Build succeeded:

Oct 30 '22 06:10 bors[bot]

It appears DappNode's sync URL does not yet support this. Is there a recommended alternative URL for DappNode?

WARN Remote BN does not support EIP-4881 fast deposit sync

Dappnode's default URL is https://checkpoint-sync.dappnode.io

Jul 27 '23 10:07 dawsbot

I've tried several urls, the main one and dappnodes and a couple others and I'm getting the same error about 4881

Aug 01 '23 19:08 icculp

@dawsbot @icculp You can safely ignore that message, it's just a warning. We're waiting on checkpoint sync providers to update their infra. The mainnet.checkpoint.sigp.io endpoint supports 4881 if you want to try it out

Aug 01 '23 20:08 michaelsproul

@michaelsproul it doesn't ignore the error for me. Idk if it's because it's for network gnosis, but it shuts down on that error #4559

Aug 01 '23 21:08 icculp

@icculp That's not the true error, there must be something else amiss. If you get an exit code 132 then you need the portable binary. If the checkpoint sync URL is timing out then you can lengthen the timeout with --checkpoint-sync-url-timeout.

Aug 01 '23 23:08 michaelsproul

Well, i am in the same situation with gnosis. I did try to add the --checkpoint-sync-url-timeout flag, but the result is still the same error:

Aug 03 09:41:05 gnosis lighthouse[382380]: Aug 03 07:41:05.374 INFO Starting checkpoint sync remote_url: https://checkpoint.gnosischain.com/, service: beacon Aug 03 09:41:05 gnosis lighthouse[382380]: Aug 03 07:41:05.492 WARN Remote BN does not support EIP-4881 fast deposit sync, error: Error fetching deposit snapshot from remote: ServerMessage(ErrorMessage { code: 500, message: "not found", stacktraces: [] }), service: beacon Aug 03 09:41:05 gnosis lighthouse[382380]: Aug 03 07:41:05.641 CRIT Failed to start beacon node reason: Error fetching finalized block from remote: ServerMessage(ErrorMessage { code: 500, message: "not found", stacktraces: [] }) Aug 03 09:41:05 gnosis lighthouse[382380]: Aug 03 07:41:05.641 INFO Internal shutdown received reason: Failed to start beacon node Aug 03 09:41:05 gnosis lighthouse[382380]: Aug 03 07:41:05.641 INFO Shutting down.. reason: Failure("Failed to start beacon node") Aug 03 09:41:05 gnosis lighthouse[382380]: Failed to start beacon node Aug 03 09:41:05 gnosis systemd[1]: beacon-chain.service: Main process exited, code=exited, status=1/FAILURE Aug 03 09:41:05 gnosis systemd[1]: beacon-chain.service: Failed with result 'exit-code'.

Aug 03 '23 07:08 fjvva

@fjvva The issue there is that the endpoint isn't returning a block:

Aug 03 07:41:05.641 CRIT Failed to start beacon node reason: Error fetching finalized block from remote: ServerMessage(ErrorMessage { code: 500, message: "not found", stacktraces: [] })

It's this CRIT (critical) log that is causing the failure, not the warning about 4881.

I asked the Gnosis devs and they're aware of the issue and will update the checkpoint.gnosischain.com server with a bugfix in the next ~24h.

In the meantime, you'd be best off trying another checkpoint sync provider, although for Gnosis there don't seem to be many. I got a different error from gateway.fm's server. Might just need to wait for the fix.

Aug 03 '23 23:08 michaelsproul

@michaelsproul thanks for the heads up. I guess Gnosis users will just have to wait a bit then.

Edit: If anyone else has the same problem, this checkpoint provider works: https://checkpoint-sync-gnosis.dappnode.io/

Aug 04 '23 06:08 fjvva

Hi there, I changed from --checkpoint-sync-url=https://checkpoint.gnosischain.com to --checkpoint-sync-url=https://checkpoint-sync-gnosis.dappnode.io and still having the same issue Failed to start beacon node, however the error is slightly different.

Aug 10 21:08:16.544 INFO Logging to file                         path: "/var/lib/lighthouse/beacon/logs/beacon.log"
Aug 10 21:08:16.544 INFO Lighthouse started                      version: Lighthouse/v4.2.0-c547a11
Aug 10 21:08:16.544 INFO Configured for network                  name: gnosis
Aug 10 21:08:16.546 INFO Data directory initialised              datadir: /var/lib/lighthouse
Aug 10 21:08:16.546 INFO Deposit contract                        address: 0x0b98057ea310f4d31f2a452b414647007d1645d9, deploy_block: 19469077
Aug 10 21:08:16.722 INFO Starting checkpoint sync                remote_url: https://checkpoint.gnosischain.com/, service: beacon
Aug 10 21:08:16.850 WARN Remote BN does not support EIP-4881 fast deposit sync, error: Error fetching deposit snapshot from remote: ServerMessage(ErrorMessage { code: 415, message: "unsupported content-type: application/octet-stream", stacktraces: [] }), service: beacon
Aug 10 21:08:16.885 CRIT Failed to start beacon node             reason: Unable to parse SSZ: OffsetSkipsVariableBytes(388). Ensure the checkpoint-sync-url refers to a node for the correct network
Aug 10 21:08:16.885 INFO Internal shutdown received              reason: Failed to start beacon node
Aug 10 21:08:16.885 INFO Shutting down..                         reason: Failure("Failed to start beacon node")
Failed to start beacon node

Aug 10 21:44:04.213 INFO Logging to file                         path: "/var/lib/lighthouse/beacon/logs/beacon.log"
Aug 10 21:44:04.214 INFO Lighthouse started                      version: Lighthouse/v4.2.0-c547a11
Aug 10 21:44:04.214 INFO Configured for network                  name: gnosis
Aug 10 21:44:04.215 INFO Data directory initialised              datadir: /var/lib/lighthouse
Aug 10 21:44:04.216 INFO Deposit contract                        address: 0x0b98057ea310f4d31f2a452b414647007d1645d9, deploy_block: 19469077
Aug 10 21:44:04.268 INFO Starting checkpoint sync                remote_url: https://checkpoint-sync-gnosis.dappnode.io/, service: beacon
Aug 10 21:44:04.726 CRIT Failed to start beacon node             reason: Unable to parse SSZ: OffsetSkipsVariableBytes(388). Ensure the checkpoint-sync-url refers to a node for the correct network
Aug 10 21:44:04.727 INFO Internal shutdown received              reason: Failed to start beacon node
Aug 10 21:44:04.729 INFO Shutting down..                         reason: Failure("Failed to start beacon node")
Failed to start beacon node

I'm using sigp/lighthouse:v4.2.0.

I appreciate your support. Regards.

Aug 10 '23 21:08 chilcano

@chilcano You need Lighthouse v4.3.0, because the Shapella hard fork has happened on Gnosis.

Aug 10 '23 22:08 michaelsproul

Thanks @michaelsproul Just updated and worked !! Thanks again. Regards.

Aug 10 '23 22:08 chilcano

I'll lock this thread now so that it doesn't continue as a generic checkpoint sync debugging thread.

If anyone else stumbles upon the EIP-4881 warning or can't get checkpoint sync to work, please open a new issue or contact us on Discord.

Aug 10 '23 22:08 michaelsproul