Conseil icon indicating copy to clipboard operation
Conseil copied to clipboard

The indexer is terribly slow due to fees processing

Open ivanopagano opened this issue 4 years ago • 0 comments

As shown by the query engine the way fees are collected takes a long time. The process is run once every N cycles of Lorre and loads past fees.

Analysis: As the number of fees grows the indexing to collect distinct values becomes useless and the engine is forced to do a linear scan. The query is: select distinct "fee", "timestamp", "cycle", "level" from "tezos"."operations" where "kind" = 'transaction' order by "timestamp" desc limit 1000

the longest amount of time is being taken up by the sort and the subsequent sequential scan, please keep in mind that kind is already an index plan is limit(.086ms) -> Unique(2.406 ms) -> sort(12,105ms) -> seq_scan(11,221ms)

Proposed solutions: Put a lower bound on the query timestamp to reduce the number of rows returned for the computation when doing the distinct operation. Forego the distinct requirement, if it's not necessary (why should it happen that we collect more than one unique fee?)

ivanopagano avatar Jun 30 '20 11:06 ivanopagano