Brooks

@HaoranYi @jeffwashington This PR is ready for re-review. Thanks!

I'll take a look. Thanks for filling this issue.

So far I have been unable to reproduce a failure with fastboot. Here's the experiments I've performed so far. For all of them I have specified `--use-snapshot-archives-at-startup when-newest` on the...

> Do you know if a panic in some thread on exit would cause this? This is what I was trying to reproduce by randomly killing the validator process in...

Ok, I've found the (an?) problem. The PR to fix it is here: https://github.com/solana-labs/solana/pull/35350

Found the other problem. Here's a GH Issue for it: https://github.com/solana-labs/solana/issues/35367.

https://github.com/solana-labs/solana/pull/35350 has been merged, so the *recovery* aspect of this failure is now fixed. Other PRs will fix the underlying issues.

I'm not familiar with these metrics/how they are used, so I've removed myself as a reviewer. I defer to the subject matter experts here.

> Hitting this would indicate we have miscalculated the alive bytes in an append vec and would be a bug in clean or shrink or append vecs. @yhchiang-sol and I...

lgtm - I'll defer a full :shipit: to Will.