osmosis
osmosis copied to clipboard
osmosis-labs
Make Export Less RAM Intensive (IAVL Changes)
Background
It currently takes VMs with 64GB of RAM to export/import full state exports. Some basic IAVL improvements can be done to improve this performance.
Suggested Design
- Key format change
- Replace all native maps with BTree maps
- We currently have atmoic writes at a per module scope, which is not efficient
- We should lower the scope of atomic writes from the 100s of MBs to the 100s of KBs
Acceptance Criteria
- Import/Export does not take extraordinary amounts of RAM to work
Can I have some questions ?
- What native maps should we replace ? Maybe maps like this ?
- We should use https://github.com/tidwall/btree for BTree maps right ?
We currently have atomic writes at a per module scope
This refers to cms Store's Write() right ? Can you explain this more ?
I am going to have to have a separate conversation with Dev to get a better / more concrete understanding of the best way forward with this. This might have to happen later this week sadly
I sadly do not have context on this, @ValarDragon would you be able to write something out in order to allow @catShaark to pick up this task?
Key Format Changes
Basically make key formats we write to disk follow write_version | path_to_node
Replace all native maps with BTree maps
This is entirely unrelated
We currently have atmoic writes at a per module scope, which is not efficient
Basically just change the batch.set & batch.writeSync usage in IAVL, to write in many smaller batches. E.g. write every 1MB atomically, don't keep growing the batches. We already have tooling for detecting when we've completed the final write, the whole thing should not be atomic, it should just be possible to safely automatically rollback / wipe latest version.
@catShaark I would love to work with you on smaller batch writing. Would you be interested in working together?
why not streaming of genesis? agree that batching would help but also streaming would be the correct solution
