Prune state trie in run-time

[peekpi]

trafficstars

Description

This pr is designed to prune the state-trie in run-time. The goal is to slow down the growth of disk usage for non-archival nodes. When a new state-trie is written to disk, it compares the old state-trie with the new state-trie, and deletes data exists in old state-trie but not in the new state-trie.

Enable state pruning

cmd line

harmony --pruning

config

[General]
  StatePruneEnable = true

TODO

• [x] 1. add more complex unit tests.
• [x] 2. make this feature optional
• [x] 3. prune in async
• [x] 4. allow pruning to be interrupted and aborted.
• [x] 5. waiting prune finished when node exit
• [ ] 6. test it on a mainnet non-archival node

[MaxMustermann2]

@peekpi do we know why state trie was intentionally dumped to disk at the end of each epoch?

[peekpi]

@peekpi do we know why state trie was intentionally dumped to disk at the end of each epoch?

I don't know, but I don't see where this feature is needed. do you know about this? @rlan35

[MaxMustermann2]

I don't know, but I don't see where this feature is needed.

Let's retain it for now. According to RJ, it is "convenient for fast state sync where only state is synced".

[peekpi]

I don't know, but I don't see where this feature is needed.

Let's retain it for now. According to RJ, it is "convenient for fast state sync where only state is synced".

How about make it to optional? And the default is to keep it.

[MaxMustermann2]

How about make it to optional? And the default is to keep it.

Let's keep the last block in epoch please.

[ONECasey]

@peekpi , is there anyway we can get an update on this PR and possibly fix the conflicts on the following files:

cmd/harmony/config_migrations.go cmd/harmony/default.go core/blockchain_impl.go internal/shardchain/shardchains.go

[peekpi]

@peekpi , is there anyway we can get an update on this PR and possibly fix the conflicts on the following files:

cmd/harmony/config_migrations.go

cmd/harmony/default.go

core/blockchain_impl.go

internal/shardchain/shardchains.go

yeah, I can fix it. But there is a problem with this pr that is difficult to solve... If two contracts have the same state, they share the same state tire. If the data of one contract changes, we cannot prune it, otherwise the data of the other contract will be missing.

So I don't think this pr can be merged unless the problem is solved.

[MaxMustermann2]

If two contracts have the same state, they share the same state tire. If the data of one contract changes, we cannot prune it, otherwise the data of the other contract will be missing.

Is there no reference counting for these states that can be used to assess whether it can be deleted? In other words, if there is a state with two references, it should not be deleted.