Feat: Shadow block recovery mechanism

[jcnelson]

This introduces the notion of "shadow blocks" and "shadow tenures" as a means of recovering from a chain stall. In the event of an irrecoverable chain stall, we'd pass an emergency SIP to declare that one or more empty sortitions actually commit to a "shadow block" that contains whatever state is needed to get the chain un-stuck. All nodes would ship with copies of these shadow blocks, and would process them the same as any other kind of block (except that they could not be mined, uploaded, or downloaded).

Leaving as a draft for now until I can expand test coverage. This touches a lot of consensus-critical code paths.

---

Here is a summary of consensus changes for shadow blocks:

• All shadow blocks MUST have a version whose highest bit is set.
• If any block in a tenure is a shadow block, then all blocks in that tenure MUST be shadow blocks (it's a "shadow tenure")
• No non-shadow blocks can be inserted into a shadow tenure.
• The block-commit of the first child tenure of a shadow block tenure MUST set parent_vtxindex to 0, which is the burnchain coinbase transaction. We assume that shadow blocks all descend from their respective PoX anchor blocks, so this will not be checked for the first child tenure. Instead, the block's parent sortition will be checked to verify that it is a shadow tenure and that the block-commit parent_vtxindex is 0 when validating the child block.
• The VRF proof of a shadow block will not be validated (since there's no seed to use)
• The miner signature of a shadow block will not be validated (since there's no miner)
• For the purposes of finding the PoX anchor block, a shadow tenure in the prepare-phase always counts as a sortition, even if there were no block-commits in that burnchain block.
• The signing weight of a shadow block is the maximal weight of the reward set. It's as if all signers signed it.
• The coinbase of a shadow block MUST be the burn address for the network. Shadow blocks will produce STX, but no one will receive it.
• The memo field of the coinbase of a shadow block MUST be the hash of the tenure-start block of the shadow block's parent tenure. This is required because shadow blocks do not have corresponding block-commits. Instead, we capture the tenure start block hash that would have gone into the block-commit as the memo field in the shadow block coinbase.
• Shadow blocks do not increase the total BTC spend for the chain
• The VRF proof for the child of a shadow block must come from the VRF seed of the shadow block -- i.e. the hash of the VRF proof in the parent shadow block's tenure's coinbase.

[jcnelson]

N.B. This PR needs to be followed by (or extend with) a PR to re-mine a sequence of blocks as shadow blocks, so we can fix a consensus bug on testnet.

We'll need more elaborate chain replay logic in the future to preserve causal relationships between affected transactions if possible. At a minimum today, we can preserve the VRF proofs.

[jcnelson]

Okay, this PR is getting kinda big. I'm going to hold off on adding the tooling for generating shadow blocks for the time being. I'll add that as a separate PR so we can move this one along.

[jcnelson]

@kantai @jferrant @obycode This is ready for re-review. I'm going to work on tooling for (quickly) synthesizing shadow blocks via stacks-inspect so we can test this on the Nakamoto testnet.

[jcnelson]

Look at that.

All of that.

All those green :heavy_check_mark:'s in CI.

Outstanding work on making them all reliable @jferrant! It drastically improves our confidence that we can make consensus-critical changes like this. Well done!

[jcnelson]

Okay, there's definitely some latent bug here in the test harness that's causing a failure. I think it may be the same one discovered by @hugocaillard in https://github.com/stacks-network/stacks-core/pull/4997.

[jcnelson]

ping @jferrant @hstove @kantai

[jcnelson]

ping @hstove @jferrant @obycode

[hstove]

@jcnelson I responded to the one unresolved comment - once that's fixed I think this is good to merge!

[jcnelson]

Thanks @hstove! Replied.

[blockstack-devops]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.