horaedb icon indicating copy to clipboard operation
horaedb copied to clipboard

Maybe we should call `mark_delete_entries_up_to` again while table finished to recover

Open Rachelint opened this issue 2 years ago • 0 comments

Describe This Problem

The flushed point in table version is updated before the deleted point's updating in wal manager.

        // Edit table version to remove dumped memtables.
        let mems_to_remove = mems_to_flush.ids();
        let edit = VersionEdit {
            flushed_sequence,
            mems_to_remove,
            files_to_add: files_to_level0,
            files_to_delete: vec![],
        };
        table_data.current_version().apply_edit(edit);

        // Mark sequence <= flushed_sequence to be deleted.
        self.space_store
            .wal_manager
            .mark_delete_entries_up_to(table_data.location(), flushed_sequence)
            .await
            .context(PurgeWal {
                table_location: table_data.location(),
                sequence: flushed_sequence,
            })?;

        Ok(())

If the database crashes behind updating in table version but before updating in wal manager. When the database restarts, the deleted point in wal manager will fall behind the flushed point in table version util new flushing is triggered. If the table's writing frequency is too low, the duration of falling behind will be too long, and it make a bad difference to wal's delayed deleting.

Proposal

Call mark_delete_entries_up_to again after recovering.

Additional Context

No response

Rachelint avatar Oct 27 '22 14:10 Rachelint