[ETH] - Blockbook Testnet Ropsten Node is out of sync when compared to Ropsten Etherscan

[lexy212]

We have a blockbook node running Testnet Ropsten but the data is out of sync when compared to Ropsten Etherscan.

For context, as of the time of writing this issue, Our node is on block 12446346 and ropsten etherscan is on block 12793048

However, our node is sync with this other explorer https://blockexplorer.one/ethereum/ropsten and the data is consistent.

From what I've been researching. I'm thinking the changes described in this blog post seems like the reason. Ropsten etherscan may be have been upgraded and integrated the Consensus Layer. (I'm not entirely sure).

Initially, our testnet node was stuck with the following errors

May 26 22:28:29 test-eth-node-01 geth[29340]: INFO [05-26|22:28:29.448] Legacy sync reached merge threshold      number=12,308,106 hash=a6fbe9..bdbf28 td=43,532,321,247,089,466 ttd=43,531,756,765,713,534
May 26 22:28:30 test-eth-node-01 geth[29340]: INFO [05-26|22:28:30.781] Legacy sync reached merge threshold      number=12,308,106 hash=a6fbe9..bdbf28 td=43,532,321,247,089,466 ttd=43,531,756,765,713,534
May 26 22:28:30 test-eth-node-01 geth[29340]: WARN [05-26|22:28:30.781] Local chain is post-merge, waiting for beacon client sync switch-over...
May 26 22:28:39 test-eth-node-01 geth[29340]: INFO [05-26|22:28:39.821] Legacy sync reached merge threshold      number=12,308,106 hash=a6fbe9..bdbf28 td=43,532,321,247,089,466 ttd=43,531,756,765,713,534
May 26 22:28:42 test-eth-node-01 geth[29340]: INFO [05-26|22:28:42.506] Legacy sync reached merge threshold      number=12,308,106 hash=a6fbe9..bdbf28 td=43,532,321,247,089,466 ttd=43,531,756,765,713,534
May 26 22:28:42 test-eth-node-01 geth[29340]: WARN [05-26|22:28:42.507] Local chain is post-merge, waiting for beacon client sync switch-over...

after I did a bit of research, I found the following issues,

• https://github.com/ethereum/go-ethereum/issues/24976
• https://github.com/ethereum/go-ethereum/issues/25118
• https://github.com/ethereum/go-ethereum/issues/25281

One Of the comments suggested to add a ttd override --override.terminaltotaldifficulty 100000000000000000000000 to the geth startup command, as described in this tweet. https://twitter.com/vdWijden/status/1529895417881800714.

i did that and it unstuck the node. As of now, our node is up and running, it still syncing data, however the data is not similar to what's on Ropsten Etherscan.

All fingers in the issues I have linked above point to running a consensus client alongside geth.

I'm asking if the blockbook team has a solution (either ready or in the works) for this. It seems like a consensus layer client has to be bundled with the blockbook software.

I'd also like to hear from anyone that has experienced this(and found a solution) or experiencing it as well.

Thanks.

[Dehumanizer77]

You need to have a consensus node running on Ropsten since some time ago, otherwise you end up on a wrong chain as you did. It's not merged into master yet, but you can build it from the eth_update branch. You can build consensus node with make deb-backend-ethereum_testnet_ropsten_consensus and you'll also need to rebuild the execution node by running make deb-backend-ethereum_testnet_ropsten as some parameters have changed there as well.

Unfortunatelly, you will have to resync both ropsten and blockbook from scratch (if you don't have ropsten blockbook with consensus layer running elsewhere):

• stop blockbook-ethereum-testnet-ropsten.service and backend-ethereum-testnet-ropsten.service
• rm -r /opt/coins/data/ethereum_testnet_ropsten/blockbook/db
• rm -r /opt/coins/data/ethereum_testnet_ropsten/backend/*
• install backend-ethereum-testnet-ropsten and backend-ethereum-testnet-ropsten-consensus packages you have buillt
• start all 3 services (blockbook, backend and the consensus backend)
• wait until everything synchronizes which can take a long time

[lexy212]

Further comparing our blockbook ropsten node with ropsten etherscan, the last block at which they both had similar data was block 12350711. Also notice that the blockhash is similar.

See screenshots below.

The next block 12350712 is where the paths diverge.

The hashes are different.

Our blockbook ropsten test node hash of block 12350712 - 0xd3a5817d40794f7131ed0c448be144cdb6eb9fec80740be8541c708d501b1f0c

Ropsten Etherscan hash of block 12350712 - 0x5010949c3acdf1d076c9e4ee8aedbb539aca48ebe19638330bc02c5b1bfb953d.

On Ropsten Etherscan. this is where start seeing information about the Consensus Info.

[lexy212]

You need to have a consensus node running on Ropsten since some time ago, otherwise you end up on a wrong chain as you did. It's not merged into master yet, but you can build it from the eth_update branch. You can build consensus node with make deb-backend-ethereum_testnet_ropsten_consensus and you'll also need to rebuild the execution node by running make deb-backend-ethereum_testnet_ropsten as some parameters have changed there as well.

Unfortunatelly, you will have to resync both ropsten and blockbook from scratch (if you don't have ropsten blockbook with consensus layer running elsewhere):

• stop blockbook-ethereum-testnet-ropsten.service and backend-ethereum-testnet-ropsten.service
• rm -r /opt/coins/data/ethereum_testnet_ropsten/blockbook/db
• rm -r /opt/coins/data/ethereum_testnet_ropsten/backend/*
• install backend-ethereum-testnet-ropsten and backend-ethereum-testnet-ropsten-consensus packages you have buillt
• start all 3 services (blockbook, backend and the consensus backend)
• wait until everything synchronizes which can take a long time

Thanks @Dehumanizer77 , I'll check this out.

[Dehumanizer77]

@lexy212 great, let me know if anything you need to help with comes up.

[lexy212]

@Dehumanizer77 I have done this for the testnet node, it's resyncing. The data size is relatively small and this should be complete in no time. Thanks once again.

I was just thinking, if it would be possible to do this without deleting the entire backend.

From what I understand, the opt/coins/data/ethereum_testnet_ropsten/blockbook/db contains blockbook indexed data, which can be deleted and easily rebuilt.

For the backend, if I run the backend (geth) in console mode,

/opt/coins/nodes/ethereum_testnet_ropsten/geth console

then using the debug.setHead() command to rewind the blockchain to start from a much earlier block i.e before the merge. This way, we can avoid deleting the entire blockchain data and the syncing should work correctly.

What I'm mostly bothered about this is that — when the merge occurs on mainnet, our node is over 4TB of data which has taken us weeks to sync [at this point in time]. It's not practical for us to delete everything and restart again just like I did for the testnet.

What do you think about this?

[Dehumanizer77]

@lexy212 even just resyncing the blockbook db on mainnet will take quite long, but I suppose the key is to implement the consensus backend before the merge so you won't have to resync anything. @martinboehm might want to confirm if I am correct...