blockchain-core icon indicating copy to clipboard operation
blockchain-core copied to clipboard

Make serialization/deserialization of ledger PoC records quicker

Open Vagabond opened this issue 3 years ago • 3 comments

We currently store a t2b'd list of t2b'd ledger_poc_v2 records. This record itself is full of binaries, so we don't preserve sub binary refernces (using more ram) and t2b over t2b seems silly.

Propose we use a real binary packing scheme to pack the binaries, and the list of the poc records more efficently so we can pull them out of the db faster with less ram usage.

Vagabond avatar Nov 30 '21 16:11 Vagabond

https://github.com/helium/blockchain-core/blob/master/src/ledger/v1/blockchain_ledger_v1.erl#L1899-L1900

https://github.com/helium/blockchain-core/blob/master/src/ledger/v1/blockchain_ledger_poc_v2.erl#L142

Vagabond avatar Nov 30 '21 16:11 Vagabond

I would propose additionally that we calculate and store the maximum height of any poc in any given poc list. This will allow us to GC without decoding, which will speed that process up immensely.

evanmcc avatar Jan 12 '22 18:01 evanmcc

By store, I mean <<Vers:1/bytes, MaxHeight:64/unsigned-native, KVList/binary>> or something similar, so that we don't need to decode the kvlist at all, just iterate over the CF and delete once the entire list is available for GC.

evanmcc avatar Jan 12 '22 18:01 evanmcc