BlockManager missing_blocks can grow unbounded leading to OOM

[SherlockShemol]

Summary

The BlockManager in consensus-core has two vulnerabilities related to missing_blocks:

1. Unbounded Growth: The missing_blocks set can grow without bound when Byzantine validators produce blocks referencing non-existent ancestors, leading to OOM.

2. Future-Flooding Attack: A naive "lowest rounds first" eviction strategy can be exploited by attackers to evict legitimate missing blocks by flooding with high-round garbage.

Problem

The code contains an explicit TODO acknowledging the first issue:

/// TODO: As it is possible to have Byzantine validators who produce Blocks without valid causal
/// history we need to make sure that BlockManager takes care of that and avoid OOM (Out Of Memory)
/// situations.

When a block references ancestors that do not exist (and cannot be fetched), these are added to missing_blocks without any upper bound. A Byzantine validator can continuously produce blocks with fake ancestor references, causing unbounded memory growth.

Additionally, a naive "lowest rounds first" eviction strategy would allow attackers to:

1. Wait until the node has legitimate missing blocks (e.g., P at Round 10)
2. Flood with garbage blocks referencing fake parents at high rounds (e.g., Round 1000)
3. When capacity is reached, the eviction strategy removes lowest rounds first
4. Result: Legitimate block P (Round 10) is evicted while garbage is retained

Impact

• Liveness Risk: Medium - Memory exhaustion OR legitimate blocks evicted
• Safety Risk: Low - Does not affect block processing correctness
• Agreement Risk: Low - Does not affect consensus decisions

Attack Vectors

Vector 1: OOM Attack

1. Attacker must be a committee member (hold signing keys)
2. Each block can reference up to committee_size fake ancestors
3. Each fake ancestor adds ~100 bytes to memory
4. At realistic attack speed (200ms/round): ~149 hours to exhaust 1GB

Vector 2: Future-Flooding Attack

1. Node at Round 10 has legitimate missing parent P (Round 10)
2. Attacker floods with blocks referencing Round 1000 fake parents
3. Naive eviction removes P first (lowest round) - Liveness failure

Proposed Fix

1. Add MAX_MISSING_BLOCKS constant (100,000 entries, ~10MB max)
2. Add gc_missing_blocks() method with distance-based eviction:
   • Define protection window around current round (±50 rounds)
   • Prioritize evicting entries FAR from current round (likely garbage)
   • Protect entries NEAR current round (likely legitimate)
3. Add capacity checks before inserting new missing blocks

Affected Code

• consensus/core/src/block_manager.rs - BlockManager::try_accept_one_block() and BlockManager::try_find_blocks()

[github-actions[bot]]

Thank you for opening this issue, a team member will review it shortly. Until then, please do not interact with any users that claim to be from Sui support and do not click on any links!

[mwtian]

Thanks for the looking into this. If it is detected on public networks that a validator is trying to overwhelm other validators by producing invalid blocks, they will get socially slashed because the behavior is detectable and easily provable. But handling this scenario in a more robust way automatically is definitely a good improvement. We will review the related PR.