rocksdb icon indicating copy to clipboard operation
rocksdb copied to clipboard

Check compaction output file sizes

Open pdillinger opened this issue 7 months ago • 0 comments

I believe that if a compaction output is somehow corrupted to have extra data after the actual RocksDB written data, we don't detect that until the DB is re-opened and you get "Sst file size mismatch: ...". We should include this sanity check on installing compaction results so that we don't end up with an un-openable DB.

Remote compactions currently have kind of the opposite problem: the file size is read from the FileSystem and treated as authoritative. So you might get a "Bad magic number" error when really the problem is an incomplete file was read/queried. We should be reporting the size mismatch by propagating the file size from the remote compaction worker.

Both kinds of compaction should be cross-checking between when was independently recorded as the size written and the size reported by the FileSystem.

pdillinger avatar May 16 '25 23:05 pdillinger