storage: using concurrent data structure to incrementally maintain storage snapshot

[skyzh]

Currently, every operation on storage snapshot will clone and create a new full snapshot, which contains a list of rowsets. Indeed, we can use some concurrent data structure to maintain this, so as to reduce the need of cloning.

https://github.com/risinglightdb/risinglight/blob/3e59c3616ffe8eeab54ea1a4164a27339a35eb4f/src/storage/secondary/version_manager.rs#L257-L265

https://github.com/ibraheemdev/seize looks like a good choice.

[BaymaxHWY]

we can use some concurrent data structure to maintain this, so as to reduce the need of cloning.

how to achieve this? make a snapshot of each epoch and only track the different from last?

[skyzh]

how to achieve this? make a snapshot of each epoch and only track the different from last?

Exactly, but I didn't come up with a detailed way to achieve this... Indeed, every operation to RisingLight's storage manifest is simply a diff operation.

[BaymaxHWY]

I'm sorry, i have no relevant experience about this. Unable to give helpful advice :) Hope i can help you in the future.

[skyzh]

No worries 🤪