Problem: no binary genesis import & export

[leejw51crypto]

currently, genesis import & export is done plain json text file. which is large & inefficient to handle.

1. add import & export with binary format
2. sqlite would be good fit , because user can also check the contents

[JayT106]

currently, genesis import & export is done plain json text file. which is large & inefficient to handle.

@leejw51crypto Could you explain in more detail the inefficiency? do you mean the import/export speed?

[JayT106]

the exporting state at height 3104880:

exporting... module genesis genutil
exported module genesis genutil, size 14, time: 29.858µs
exporting... module genesis auth
exported module genesis auth, size 255533801, time: 18m19.766358778s
exporting... module genesis vesting
exported module genesis vesting, size 2, time: 2.002µs
exporting... module genesis bank
exported module genesis bank, size 63231917, time: 9m30.362255526s
exporting... module genesis capability
exported module genesis capability, size 1118, time: 6.782134ms
exporting... module genesis crisis
exported module genesis crisis, size 50, time: 182.056µs
exporting... module genesis gov
exported module genesis gov, size 2832, time: 10.035674ms
exporting... module genesis mint
exported module genesis mint, size 312, time: 241.683µs
exporting... module genesis slashing
exported module genesis slashing, size 7081945, time: 4m4.39138446s
exporting... module genesis distribution
exported module genesis distribution, size 33976, time: 1.175575771s
exporting... module genesis staking
exported module genesis staking, size 33249, time: 475.912321ms
exporting... module genesis upgrade
exported module genesis upgrade, size 2, time: 1.723µs
exporting... module genesis evidence exported module genesis evidence, size 15, time: 43.585µs exporting... module genesis ibc exported module genesis ibc, size 297076853, time: 25m11.473207169s exporting... module genesis params exported module genesis params, size 0, time: 1.256µs exporting... module genesis feegrant exported module genesis feegrant, size 6978, time: 56.168646ms exporting... module genesis authz exported module genesis authz, size 876, time: 229.711µs exporting... module genesis transfer exported module genesis transfer, size 3140, time: 5.443416ms exporting... module genesis evm exported module genesis evm, size 4967554826, time: 17h16m30.357516549s exporting... module genesis feemarket exported module genesis feemarket, size 202, time: 36.02913ms exporting... module genesis cronos exported module genesis cronos, size 6018, time: 424.298829ms appExporter time 18h27m23.268923683s

starting marshal genesis done marshal genesis, time 2m32.038029189s, size: 5590573903 starting marshal sort genesis done marshal sort genesis, time 5m43.869593346s, size: 5590573903 exporting Binary exported Binary, size: 5590573903

[JayT106]

The long-term solution might be using the ORM feature after Cosmos SDK v0.46.x. However, it will need the whole module's state migration using the ORM table and the known downgrading DB performance issue.

Currently trying to export the genesis state to files in each module and save it to a genesis/[module] folder. Also, splits the state into several files if the state is huge, like evm, ibc, auth. Therefore, it can save the memory requirement when doing the import/export. The current genesis export consumes more than 64GB RAM to be able to finish. WIP

[JayT106]

testing the genesis export to genesis/[module], it shows we can reduce some space and time usage with the same dataset.

Time: 15h56m v.s. 18h27m Space: 4.6G v.s. 5.6G

For the OOM issue, it relates to https://github.com/tendermint/tm-db/issues/272 , we will need to limit the file open size to around 1000

[JayT106]

submit PR in the SDK, but it also requires the other project to implement the methods. https://github.com/cosmos/cosmos-sdk/pull/13032