Optimize the block resyncing

[Jim8y]

Summary or problem description Currently, when synchronizing the blockchain, we need to verify the signature of each block to ensure data authenticity and integrity. This process is extremely time-consuming, requiring verification of numerous signatures. We aim to optimize this process to improve synchronization efficiency.

Do you have any solution you want to propose? We propose implementing a selective verification mechanism:

1. Perform full signature verification only for blocks that change consensus nodes, ensuring the authenticity of these critical blocks.
2. For other blocks, use the hash of the previous block (prehash) to verify their authenticity, without conducting full signature verification.
3. This approach can significantly reduce computational overhead during the verification process while maintaining the overall security and integrity of the blockchain.

To ensure the correctness of the updated consensus nodes, we hardcode the consnesus address history in the neo core.

Where in the software does this update applies to?

• Consensus
• Ledger
• Network Policy
• P2P (TCP)

[roman-khimov]

I don't see how this can work at all. And consensus history has nothing to do with block correctness. The last point from #3463 is more viable as a shortcut, but it's nothing on its own, proper state sync should be available first. And then people should always have an ability to check the full chain if they'd like to. Right now it's about 1h of time. Not a small number, but not a very big one either.

[Jim8y]

this is for a verified "noverify". first of all, its not 1 hour in c#, far from it, second, even if its 1 hour, why cant we make it 50 minutes

[shargon]

I think that is good to verify only when consensus changed (only in bulk sync), but if it's wrong, we need a checkpoint

[roman-khimov]

If you trust blocks you're importing (like you really trust that dump file), you can skip signature verification completely and hashes are to be checked anyway (see SkipBlockVerification in NeoGo). But if you don't, you have to check everything, because anyone can create a block with the same NextConsensus as current (or past) one and invalid signatures. I don't see how checking here and not checking there makes it better, it's either you trust input data or not to me (and the default of course is not trusting).

[Jim8y]

checkingpoints as @shargon said is the solution.

[shargon]

But you currently can import with noVerify, isn't it?

[Jim8y]

But you currently can import with noVerify, isn't it?

yes, this issue is trying to have a solution to have the speed of "noverify", while make it verifiable.

[shargon]

But you currently can import with noVerify, isn't it?

yes, this issue is trying to have a solution to have the speed of "noverify", while make it verifiable.

We can create an enum, { verify, noVerify, verifyEpoch }

[cschuchardt88]

What If we use previous block hash to sync the clients data. After all is synced, then there should be full verify of the blockchain (process starts). This process shouldn't stop new blocks from syncing or coming in from network. As long as the hashes meet up. Once full verify sync catches up.

Processes:

1. Fully sync by previous block hash the whole chain.
   • If hashes don't match pull next block from the network. (Then continue using local storage.)
2. Run over the whole chain again, but with full verify.
   • Starts after process 1 has completed.
   • MUST NOT stop new blocks from syncing or coming in from network. (If process 1 has completed.)
   • If user tries to send or view transactions there should be a warning screen stating that data integrity checks HAVE NOT completed and on-chain data may be different. "Send now or Send when sync is complete" (queuing outbound transactions).
   • Note: this is the long process that takes a long time. At least users can still interact with local chain data.