Kosyrev Serge

This might be a relevant effort: https://github.com/input-output-hk/ouroboros-network/tree/nfrisby/CAD-4214-forge-times-via-db-analyser cc @JaredCorduan @lehins

By estimation, this seems like it will become a necessary component of large-state CI benchmarks that we do -- in all potential future paths. So this should be higher priority.

run `2022-06-01-18.53.1.35.0.9823.k51-5ep-360kTx-4000kU-1000kD-64kbs` on AWS

1. point of error: https://github.com/input-output-hk/cardano-node/blob/master/bench/locli/src/Cardano/Analysis/MachTimeline.hs#L324

~The logic of error here is that we're seeing a `LOTraceLeadershipDecided` before `LOTraceStartLeadershipCheck` -- which indeed _is_ weird, if true and if `locli` didn't trip over itself in some way.~...

Whether it's true or not, can be determined from the source of truth -- the logs from the particular node -- however the error message is unhepfully not mentioning the...

Ok, so we've ran the new `locli` on an old run, which we're 100% sure has all log messages proper -- and we've got the same failure. So it is...

Ok, giving `--dump-logobjects` to `wb a` will give us insight on how `locli` views the stream of logobjects: `wb --cls a --filters size-small --dump-logobjects std 2022-05-09-15.24.1.34.1.36a3.k51-7ep-14kTx-4000kU-1000kD-73kbs-12MUTx-10BStTx-50MUBk-40BStBk-1i-1o--null`

Ok, so we know it's the `performance-analysis` portion of the analysis that fails (aka `MachPerf` in haskell speak). Next thing -- iterating over all the gigabytes of logs is slow,...

What does the error message say? `LeadDecided for noncurrent slot=SlotNo 4068 cur=SlotNo 4069` What does it mean, on the most basic level? It means, that we've encountered a `LOTraceLeadershipDecided` with...