investigate large db size

[boz]

Testnet's database ballooned to an alarming size:

6.9G	data.testnet/akashd/data/tx_index.db
1023M	data.testnet/akashd/data/cs.wal
2.3G	data.testnet/akashd/data/state.db
5.0G	data.testnet/akashd/data/blockstore.db
40K	data.testnet/akashd/data/evidence.db
30G	data.testnet/akashd/data/application.db
45G	data.testnet/akashd/data
268K	data.testnet/akashd/config
45G	data.testnet/akashd

• [ ] How much space does our various models take up in the database? How much of the data is the index (key), how much is the value?
• deployment.Deployment
• deployment.Group
• market.Order
• market.Bid
• market.Lease
• [ ] make some estimations looking at data_size(lease_count)
• [ ] explore some options for reducing db size
• uint IDs instead of large compound keys?
• is any data present in both index and value?
• [ ] any tools we can write or use to monitor/analyse database?
• can we expose (prometheus) metrics?

Related to #604 - adding more indexes may increase unless we re-type our keys (use uints).

[anilcse]

@akhilkumarpilli can you add the observations here.

[akhilkumarpilli]

Ran single-node testnet on two instances to investigate db size. We ran around 89K blocks.

Node with pruning syncable has application db with size 4.2 MB. Whereas node with pruning nothing has application db with size 328 MB.

We can see the clear difference on application db size. pruning syncable (default) has better optimization and pruning nothing will use in a lot of disk.

[lightiv]

For our Validators, why are we running them with --pruning nothing vs. --pruning syncable? Can we reset the database and then restart with --pruning syncable?

My VPS is at 72% disk usage. It's a good thing we are about to shutdown the validators for phase 2 as I estimate that I have 2 to 3 weeks before running out of storage space.

[anilcse]

@lightiv we have state load issues with other pruning strategies: https://github.com/cosmos/cosmos-sdk/issues/5746

[hydrogen18]

Do we consider this solved with the latest network upgrade @boz ?