witnet
Read block_chain from database instead of from memory
After moving the UTXO set away from memory (#2159), the biggest memory usage is from the block_chain field from ChainState, which is a map from block epoch to block hash:
https://github.com/witnet/witnet-rust/blob/c3b1d86ca229ebf8fe5651bfb35d8db9c902eb54/data_structures/src/chain.rs#L3189
Given that we now have more than 1 million blocks, this data structure should consume around 36MB of RAM, but in practice it is closer to 70MB.
The implementation should be exactly the same as moving the UTXO set, or even easier because we never remove consolidated blocks, however there is one impediment: we probably need to be able to iterate the database in order, forward and backwards. This is because we are using the range method from BTreeMap:
https://doc.rust-lang.org/std/collections/struct.BTreeMap.html#method.range
to solve queries such as "give me the last block hash before epoch N", or "give me all the blocks between epoch 100 and 110". While the second query could be solved by checking all the possible blocks individually (100, 101, 102, ...), it will probably be more performant to use a sorted iterator directly.
I opened a PR with two new methods for the Storage trait: prefix_iterator_forward and prefix_iterator_reverse: #2171 , which should be enough to implement this use case.
Did the changes implemented following #2171 solve this? Or are we ready to resume work on this item?
I don't remember, but I believe there was a small issue with that pull request, related to the node expecting iterators to implement DoubleEndedIterator (and being able to call .rev() on them). As mentioned in that pull request, a good starting point would be to implement a method like iterate_range(start: &[u8], end: &[u8]) -> DoubleEndedIterator in the Storage trait.