Investigate why snapshotting reported warning about non-existent key on Pontus

[yangzhe1990]

Background Snapshott cfxcore::sto - In MptCursorRw, non-existing key ... is asked to be deleted.

From storage point of view, the only cause of this behavior is that a key is asked to be deleted without checking if it's already empty.

But from VM perspective, the only reasonable cause is that a contract deletes a storage key which did not exist.

We should make sure that it is due to this cause and It should have a reasonable explanation to this behavior.

[yangzhe1990]

So far all reported keys are contract storage keys.

An example 2020-05-08T14:34:56.503728535+02:00 WARN Background Snapshott cfxcore::sto - In MptCursorRw, non-existing key [136, 81, 52, 212, 29, 92, 85, 241, 46, 178, 54, 164, 50, 74, 225, 27, 246, 119, 217, 78, 100, 97, 116, 97, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2] is asked to be deleted.

The key is 56 bytes, first 20 bytes is the account address, the last 32 bytes is the vm key, "100, 97, 116, 97" in the middle denote "data" subtree.

[yangzhe1990]

I'm running a debug build to verify whether there is a bug or not, then close this issue

[fanlong]

It sounds like a potentially serious issue we need to investigate. I raise it to top priority

[yangzhe1990]

The behavior seems expected, but I'd like to introduce a change, to prevent deleting a non-existent key when committing an epoch. This could reduce 90% of the complain in the log.

The rest 10% being a key set but later deleted in different epoch within the same snapshot range. I think this is acceptable, and we can introduce some debug code to monitor these 10% cases.

[yangzhe1990]

Action item: a change, to prevent deleting a non-existent key & prevent committing unchanged data when committing an epoch @yangzhe1990 or @zimpha