cberner
Avoid recovery when using Durability::Paranoid
Currently, redb sets the needs_recovery flag as soon as you open the database, which means any unclean shutdown will trigger recovery. For large databases, this isn't a great experience -- it means a simple Ctrl-C can end up locking you out of your database for a half-hour or more while it recovers. It would be really nice if there were a way to avoid this.
Of course, recovery is unavoidable after an unclean shutdown when using Durability::Immediate, since there's no way to know whether the last commit was complete. But if you're using Durability::Paranoid, it seems like recovery shouldn't be necessary! This would let applications opt into slower commits in exchange for not needing recovery, which would be a big usability win when the database is large.
To make this work, I believe redb would need to write out the allocator state as part of Durability::Paranoid commits (or this could be a new durability level above Paranoid, to avoid slowing down existing Durability::Paranoid users who don't care about recovery speed). But I think it could be done in a fully backwards- and forwards-compatible way.
Does this sound right to you? And if so, do you think it would be a worthwhile change?
Ya, that seems useful. It should probably be separate from Paranoid since the allocator state grows linearly in the size of the database, and could be quite large.
Do you think a new Durability level, or a separate Database::flush(&self) method is more idiomatic?
Either way works, but I think a new Durability level would be nicer. This minimizes the performance impact (since calling Database::flush() would mean an extra fsync(), whereas a new Durability level lets us do it as part of the regular two-phase commit), and it also avoids having a gap in between commit() and Database::flush() where a crash would still require recovery.
If it's useful, I can describe the design I had in mind -- I haven't implemented it yet, but I think I have a decent idea of how it would work.
Ya, a new Durability level seems good to me. Happy to merge a PR if you want to implement it!
Awesome! I'll try implementing this and I'll send you a PR in a week or two.
I'm just finishing up the code here; I'll have a PR for you soon! In the meantime, I've also got a few small fixes for preexisting redb bugs that I ran into -- I'm going to submit those first, because it'll make testing the main PR easier.