ConsensusErrorException: read-txPrev-failed-insufficient

[sondreb]

Problem

In the latest code base there is an problem that causes an ConsensusErrorException to be thrown on every single block after the latest checkpoint has been reached.

Result

Sync ends up becoming very slow, due to massive amount of exceptions thrown.

How to reproduce

1. Get latest source.
2. Remove all checkpoints after block 4000.
3. Add breakpoint to line 45 in ConsensusErrorException.cs
4. Run with debugging and wait for sync to reach 4000. You might receive one or two similar exceptions before block 4000 is reached.

After last checkpoint (4000) has been reached, this throw will be raised on every (or almost every) block and cause massive amount of exceptions to be thrown, resulting in very slow chain sync.

Exception details:

Code: "read-txPrev-failed-insufficient"
Message: "read txPrev failed insufficient information"

Call stack

Here is the source code that causes this exception to be raised.

ProvenheaderCoinstakeRule.cs

The above was called by this:

Which was again called by this:

[dangershony]

This is actually by design.

Notice we mark the flag InsufficientHeaderInformation to true, this means we do not have enough information to validate the header and it will be put in a cache temporary until we believe we have more information (in practice we retry every X interval, the logic is specified in the ConsensusManagerBehavior.ConsensusTipChangedAsync method).

see form the comment in the class

// Try cache consumption every N blocks advanced by consensus. // N should be between 0 and max reorg. 20% of max reorg is a good random value. // Higher the N the better performance boost we can get.

Why we do this?

A proven header contains the transactions output that found the block (that way we can check that the input of that output is in fact valid) however to be able to check the input is valid we need to actually have validated the transaction that input belongs or more correctly the the block that transaction was mined in, if the header we received is too far in the future (in case we are in IBD for example) we may not have validated the block yet that the stake of that header is spending so we mark the header as insufficient and try it again later once we validated some more blocks.

This slows down header fetching in IBD, we will never be able to reach the tip of the chain when downloading header because we need to validate blocks in the processes in order to be able to validate more headers.

How to speed up IBD Using checkpoints we bypass the proven headers check (those headers are trusted by the client so we don't need to validate the inputs)

Dynamic Checkpoints There is a proposed PR #2972 to allow specifying a dynamic checkpoint in config. That means the initial sync is trusted, a user will find a header that is correct in a few public explorers and specify the checkpoint in configuration, the node will then sync much faster to that header.

[MithrilMan]

@dangershony what about AssumeValid? AFAIK bitcoin core dropped the use of checkpoints since 2014 using AssumeValid instead, even if this was kind of against satoshi vision that planned to have new checkpoint(s) at every new release.

the main difference is that checkpoints force consensus (so you need to have these blocks in your chain) while assumevalid allow a reorg to replace those assumed valid blocks

we could already use technically assumevalid somewhat like dynamic checkpoints

https://github.com/bitcoin/bitcoin/blob/452bb90c718da18a79bfad50ff9b7d1c8f1b4aa3/doc/release-notes/release-notes-0.14.0.md#introduction-of-assumed-valid-blocks

[dangershony]

I would not mix checkpoints with AssumedValid for the reason you specified (and to be conservative, we don't know what vulnerabilities we might introduce by mixing AV with PH) and AssumedValid just skips validation it does not enforce a specific chain.

Dynamic Checkpoint is really just another word to Configuration Based Checkpoints