Stalls while syncing to tip

[brice-stacks]

While performing a genesis sync, my node starts to randomly stall at some point. When this happens, it will not advance for 20-60 minutes, then it will suddenly start advancing again. Restarting the node will also kick-start it.

In the logs, I see a few things that look suspicious:

1. Peers are sending current blocks when this node is not yet caught up to the tip. In the logs below, the node is currently at block 718,755, but the incoming block is 4,553,914

Nov 03 10:54:15 obynuc stacks-node[1552486]: INFO [1762185255.992613] [stackslib/src/net/relay.rs:901] [relayer:20444] Handle incoming Nakamoto block dd441ff8cf1bb92f1e020dd9a125212cc9360600/366823d570023cea7fc53d1cc54bfb7228c121c43daacaa84c1a61d26f0b1203 obtained via Pushed, block_id: 8df1946514943821c78595051065236194db4c9d84a55f6f72e45dc33787d185
Nov 03 10:54:15 obynuc stacks-node[1552486]: WARN [1762185255.992749] [stackslib/src/net/relay.rs:1741] [relayer:20444] Could not process pushed Nakamoto block 8df1946514943821c78595051065236194db4c9d84a55f6f72e45dc33787d185: DBError(NotFoundError)

2. The following error always shows up before the stalls. That block is just a little bit ahead of the node's current tip. Eventually, the node begins to receive the correct Stacks blocks again and continues advancing its tip.

Nov 03 10:54:19 obynuc stacks-node[1552486]: INFO [1762185259.414680] [stackslib/src/chainstate/nakamoto/coordinator/mod.rs:1073] [chains-coordinator:20443] Process burn block 888651 reward cycle 106 in /var/lib/stacks/data/mainnet/burnchain, in_prepare_phase: false, is_rc_start: true, is_prior_in_prepare_phase: true, burn_block_hash: 00000000000000000000da506f7b0672affd0d4ae134e7132345bc89b6b63e2f
Nov 03 10:54:19 obynuc stacks-node[1552486]: INFO [1762185259.427230] [stackslib/src/chainstate/nakamoto/coordinator/mod.rs:517] [chains-coordinator:20443] No PoX anchor block known yet for cycle 106
Nov 03 10:54:19 obynuc stacks-node[1552486]: ERRO [1762185259.443228] [stackslib/src/chainstate/nakamoto/coordinator/mod.rs:739] [chains-coordinator:20443] Missing canonical anchor block

3. I see lots of failures from the StackerDB in the logs. The errors make sense, because the current signer set is likely to be different than the signer set of wherever this node is currently synced to. Could we just ignore the stackerdb completely while we are still syncing?

Nov 03 11:45:02 obynuc stacks-node[1552486]: WARN [1762188302.747490] [stackslib/src/net/relay.rs:2426] [relayer:20444] Failed to store chunk for StackerDB, stackerdb_contract_id: SP000000000000000000002Q6VF78.signers-1-1, slot_id: 11, slot_version: 4676890, num_bytes: 179, error: Bad DB slot signer (SP1EZ3CW1HXXP4S2C0SM3D2N5PFSHJ9BTN9BZHT5D,11)