bookkeeper icon indicating copy to clipboard operation
bookkeeper copied to clipboard

[improve][server]Add entry digest in journal log file

Open oneby-wang opened this issue 4 months ago • 1 comments

Hi, I recently read BookKeeper journal source code, and I found that BookKeeper journal didn't use any digest to guarante journal data's integrity and consistency. https://github.com/apache/bookkeeper/blob/2789316c18e12cbb6d17fa4a023410dbad6593a0/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Journal.java#L811-L885 Due to the lack of digest validation, we don't know whether the journal log data is corrupted or not. So we can only just do replay scan until while loop breaks out or an IOException happens. https://github.com/apache/bookkeeper/blob/2789316c18e12cbb6d17fa4a023410dbad6593a0/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Journal.java#L822-L882 If journal data is corrupted(disk data corruption or modified by mistake), and scanning incorrect journal log data happens to produce some incorrect entries(entris that already exists in this bookie), the old entries will be replaced both in RocksDB and entryLog file, which may cause data loss. https://github.com/apache/bookkeeper/blob/2789316c18e12cbb6d17fa4a023410dbad6593a0/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieImpl.java#L598-L607 ZooKeeper txnLog will append a digest after a record. So I think BookKeeper also need to do that. https://github.com/apache/zookeeper/blob/9e8296f678a9de8d9c30e3bc086a79c0cd39ef65/zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/FileTxnLog.java#L276-L327

oneby-wang avatar Aug 28 '25 15:08 oneby-wang

If BookKeeper client set ledger disgest type to DummyDigestManager, old entry data will be overwritten. If BookKeeper client set entry disgest type to other type, eg CRC32C, CRC32, old entry data will be dropped by client due to digest failure.

oneby-wang avatar Aug 28 '25 23:08 oneby-wang