Error signing attestation - Corruption: bad entry in block

[twoeths]

Describe the bug

Got this errors all the time in a holesky node which cause all attestations missed

Apr-17 02:01:37.392[]                error: Error signing attestation slot=1451408, index=16, head=0x907e5c902fb7bf0bd33614967aa327ebff1b1b29a660318e81cbba4e539bab75, validatorIndex=1535375 - Corruption: bad entry in block
Error: Corruption: bad entry in block
Apr-17 02:01:37.392[]                error: Error signing attestation slot=1451408, index=53, head=0x907e5c902fb7bf0bd33614967aa327ebff1b1b29a660318e81cbba4e539bab75, validatorIndex=1535410 - Corruption: bad entry in block

this is caused by https://github.com/google/leveldb/issues/704 also notice that the error is recorded in vc_slashing_protection_attestation_errors_total metric

Expected behavior

• The error was gone when I make a fresh validator-db folder and restart vc again, so we should add this to troubleshooting part of lodestar documentation
• This error comes from leveldb and throw all the way to "signing attestation" flow which make it confusing. We should find a way to signal that's an error coming from leveldb, for example Database error - Corruption: bad entry in block. One way is to wrap with an error in @lodestar/db package for all apis

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.17.0

[philknows]

Adding some previous context from @wemeetagain on Discord:

it looks like a leveldb error node_modules/leveldown/deps/leveldb/leveldb-1.20/table/block.cc: status_ = Status::Corruption("bad entry in block"); not sure why that would happen tho likely ez "fix" just to get the host back up and running would be to wipe the db but to diagnose the cause, we need to keep a copy of the corrupd db and also analyze the logs, possibly also the system logs