Write an anti "write amplification" layer to avoid running the same ops repeteadly

[iduartgomez]

Due to broadcasting and forwarding mechanisms is very possible nodes may be hit several times, from different peers, to replicate the same exact operation in the case of update notifications.

The solution to avoid duplicate costly work (like executing the same contract with the same value over and over) and be safe that this will apply to any ops is to keep a commit log for transactions, since the history of an initial operation is covered by the span of the same id, and double check that this operation has not been already commited previously and resolved.

The commit log can be clean up when garbage from old ops who have timed out is cleaned up based on this same time out duration.

[sanity]

Could we solve this by storing hashes of recently received updates and disregarding duplicates?

[iduartgomez]

I will check the exact details when I fix this but a combination of both things is probably the most optimal: id + hash of the op; so you can shard by transaction id and then deduplicate by hash (instead of having to compare every transaction).

The commit log can be of the hashes. It may be unnecessarily expensive to do that though since we can embed (and should do) emit time of the individual and that + the id may be enough.

There are certainly several ways this can be solved. Openned the issue so I don't forget it has to be fixed.

[github-actions[bot]]

Pivotal Tracker story: https://www.pivotaltracker.com/story/show/184207118

[iduartgomez]

Completed in #877 and #878