wormhole
wormhole copied to clipboard
wormhole-foundation
skipped or unavailable block retrieved on retry attempt errors
2021-11-16T10:20:23.160Z WARN wormhole.root.solwatch-finalized solana/client.go:378 SOLANA BUG: skipped or unavailable block retrieved on retry attempt {"empty_retry": 2, "slot": 107151303, "commitment": "finalized"}
2021-11-16T10:20:23.160Z INFO wormhole.root.solwatch-finalized solana/client.go:294 fetched block {"slot": 107151303, "num_tx": 949, "took": 0.270551497, "commitment": "finalized"}
2021-11-16T10:20:22.889Z DEBUG wormhole.root.solwatch-finalized solana/client.go:237 requesting block {"slot": 107151303, "commitment": "finalized", "empty_retry": 2}
2021-11-16T10:20:17.889Z INFO wormhole.root.solwatch-finalized solana/client.go:256 empty slot {"slot": 107151303, "code": -32004, "commitment": "finalized"}
2021-11-16T10:20:17.871Z DEBUG wormhole.root.solwatch-finalized solana/client.go:237 requesting block {"slot": 107151303, "commitment": "finalized", "empty_retry": 1}
2021-11-16T10:20:12.871Z INFO wormhole.root.solwatch-finalized solana/client.go:256 empty slot {"slot": 107151303, "code": -32004, "commitment": "finalized"}
2021-11-16T10:20:12.863Z DEBUG wormhole.root.solwatch-finalized solana/client.go:237 requesting block {"slot": 107151303, "commitment": "finalized", "empty_retry": 0}
2021-11-16T10:18:26.581Z INFO wormhole.root.solwatch-confirmed solana/client.go:180 fetched current Solana height {"commitment": "confirmed", "slot": 107151303, "lastSlot": 107151303, "pendingSlots": 0, "took": 0.013463249}
2021-11-16T10:18:25.827Z INFO wormhole.root.solwatch-confirmed solana/client.go:294 fetched block {"slot": 107151303, "num_tx": 949, "took": 0.257711717, "commitment": "confirmed"}
2021-11-16T10:18:25.570Z DEBUG wormhole.root.solwatch-confirmed solana/client.go:237 requesting block {"slot": 107151303, "commitment": "confirmed", "empty_retry": 0}
2021-11-16T10:18:25.570Z INFO wormhole.root.solwatch-confirmed solana/client.go:180 fetched current Solana height {"commitment": "confirmed", "slot": 107151303, "lastSlot": 107151300, "pendingSlots": 3, "took": 0.01062766
Related:
https://github.com/certusone/wormhole/blob/c1502bce136b79c4769d929fd07971c771924221/node/pkg/solana/client.go#L260-L267
Possibly related to:
- https://github.com/solana-labs/solana/pull/20407
@leoluk can you double check if this is still relevant and reopen it if so?
This still happens. I see guardiand 2.23.2 printing
SOLANA BUG: skipped or unavailable block retrieved on retry attempt
a lot, it fluctuates between several times per minute to several times per second.
I don’t know what Guardiand does, but we have some internal tools that import Solana blocks. Solana’s RPC API is limited in that it cannot tell you the difference between “I know for sure this slot was skipped” vs. “A block might exist for this slot but I don’t have it locally”. We work around it by querying the range of slots that the RPC is supposed to have using get_slot and minimum_ledger_slot, then shrinking the range by a big margin on both sides to account for hitting different nodes that may be slightly out of sync or that might have pruned at different times, and then for the remaining range we call get_blocks. If a slot is missing there, then we assume it was really skipped. So if the current finalized slot is X, then we’d only mark slots before X – Y as skipped, and for slots in [X – Y, ..., X] we treat them as unknown and retry later, for some margin Y. We set Y = 750 slots.
