`Missing data in Order` extremely pervasive

[helins]

During a 10 min simulation, a single peer can log it hundreds of thousands of times, commonly looking after the same hash several times. This is bound to have a very negative impact on performance.

[helins]

E.g. Counts from peer log files based on latest develop:

2.cvx:2
3.cvx:160
4.cvx:87
7.cvx:2
5.cvx:85
1.cvx:110759
0.cvx:2
8.cvx:34736
9.cvx:424428
6.cvx:495000

[mikera]

At some point we will just turn of the logging (or drop it down to DEBUG level at least). A Peer must be robust to this error (an attacker could deliberately forge an order with missing data, for example). The real problem to solve is why it is so frequent under normal-ish conditions.

[helins]

Tested 7 full runs against 80589055 and there seems to be some improvement but hard to tell exactly to what extent. Some runs didn't have those, whereas they were systematic previously. But other runs look as usual, e.g.:

log/peer/1.cvx:49
log/peer/0.cvx:3
log/peer/4.cvx:30296
log/peer/5.cvx:34202
log/peer/8.cvx:105950
log/peer/3.cvx:120074
log/peer/9.cvx:187928
log/peer/7.cvx:200776
log/peer/2.cvx:202076

[mikera]

Will continue investigating this one. The network should be robust to these, but happening this frequently is a concern (and potential significant performance loss)