tantivy
tantivy copied to clipboard
quickwit-oss
one collector per agg request instead per bucket
In this refactoring a collector knows in which bucket of the parent their data is in. This allows to convert the previous approach of one collector per bucket to one collector per request.
Add PagedTermMap as another TermAggregationMap to reduce memory usage compared to a HashMap
It contains an optimization for low cardinality bucket id Remove Clone on the collector (we only have one instance now)
Future Work
- Fetch all values for all buckets once per collector (currently each collect fetches their own data per bucket)
- Improve perf of group by bucket id in caching layer
- Improve low cardinality detection
- Remove
PerRequestAggSegCtx, we can store now everything in the collector
Performance
The heavy hitters are drastically reduced in terms of memory and CPU. For term aggs with many terms, we use a lot less memory. We use some more buffers to pass docs, which increases memory consumption for some aggs.
Biggest regression is
terms_zipf_1000_with_avg_sub_agg Avg: 9.1580ms (+44.85%)
Which should be fixed when we fetch all values for all buckets at once.
full
average_u64 Memory: 22.0 KB (-0.16%) Avg: 2.9744ms (-2.11%) Median: 2.9667ms (-1.72%) [2.9211ms .. 3.0892ms]
average_f64 Memory: 22.0 KB (-1.08%) Avg: 3.1009ms (-2.39%) Median: 3.1093ms (-2.05%) [3.0184ms .. 3.1938ms]
average_f64_u64 Memory: 24.5 KB (+4.16%) Avg: 5.7544ms (-2.11%) Median: 5.7423ms (-2.00%) [5.6601ms .. 5.9445ms]
stats_f64 Memory: 22.3 KB (+0.16%) Avg: 3.1125ms (-2.11%) Median: 3.1107ms (-1.75%) [3.0307ms .. 3.2594ms]
extendedstats_f64 Memory: 23.6 KB (+5.41%) Avg: 3.2828ms (-0.06%) Median: 3.2812ms (+0.66%) [3.2280ms .. 3.3534ms]
percentiles_f64 Memory: 39.7 KB (+38.89%) Avg: 7.2385ms (+1.42%) Median: 7.2009ms (+1.30%) [7.1505ms .. 7.6902ms]
terms_7 Memory: 36.9 KB (+3.54%) Avg: 2.3685ms (+2.30%) Median: 2.3561ms (+2.10%) [2.3277ms .. 2.5662ms]
terms_all_unique Memory: 14.7 MB (-50.24%) Avg: 7.4296ms (-56.52%) Median: 7.3689ms (-56.78%) [7.1606ms .. 7.9323ms]
terms_150_000 Memory: 3.0 MB (-55.96%) Avg: 6.4653ms (-34.01%) Median: 6.4128ms (-34.09%) [6.1846ms .. 6.9446ms]
terms_many_top_1000 Memory: 5.2 MB (-33.38%) Avg: 9.5873ms (-26.15%) Median: 9.5881ms (-25.71%) [9.2518ms .. 10.2681ms]
terms_many_order_by_term Memory: 3.0 MB (-55.96%) Avg: 5.0931ms (-55.92%) Median: 5.0741ms (-56.18%) [4.8934ms .. 5.3915ms]
terms_many_with_top_hits Memory: 50.0 MB (-11.50%) Avg: 107.1471ms (-29.63%) Median: 107.0600ms (-29.54%) [102.1220ms .. 113.0843ms]
terms_all_unique_with_avg_sub_agg Memory: 56.7 MB (-39.11%) Avg: 17.7848ms (-73.68%) Median: 17.6780ms (-73.85%) [17.1143ms .. 19.1651ms]
terms_many_with_avg_sub_agg Memory: 13.5 MB (-34.50%) Avg: 15.5057ms (-42.98%) Median: 15.3836ms (-43.13%) [15.0912ms .. 16.7996ms]
terms_status_with_avg_sub_agg Memory: 102.5 KB (+67.11%) Avg: 6.3254ms (+16.33%) Median: 6.2607ms (+15.06%) [6.2058ms .. 6.7582ms]
terms_status_with_histogram Memory: 137.9 KB (+28.77%) Avg: 6.1406ms (+15.09%) Median: 6.0987ms (+14.60%) [6.0394ms .. 6.5064ms]
terms_zipf_1000 Memory: 69.4 KB (-12.61%) Avg: 2.2273ms (+6.71%) Median: 2.2202ms (+6.72%) [2.1988ms .. 2.3291ms]
terms_zipf_1000_with_histogram Memory: 1.2 MB (+20.46%) Avg: 23.3941ms (+4.34%) Median: 23.2952ms (+3.94%) [23.0893ms .. 23.9924ms]
terms_zipf_1000_with_avg_sub_agg Memory: 464.6 KB (+27.68%) Avg: 9.1580ms (+44.85%) Median: 9.1284ms (+44.70%) [8.8603ms .. 9.7444ms]
terms_many_json_mixed_type_with_avg_sub_agg Memory: 20.6 MB (-20.71%) Avg: 26.1827ms (-38.31%) Median: 25.9265ms (-38.81%) [25.1805ms .. 27.8912ms]
cardinality_agg Memory: 3.7 MB (-0.00%) Avg: 29.1323ms (-0.77%) Median: 29.1288ms (-0.62%) [28.6304ms .. 30.0201ms]
terms_status_with_cardinality_agg Memory: 5.5 MB (+0.79%) Avg: 70.5721ms (+1.07%) Median: 70.1373ms (+0.61%) [69.4065ms .. 74.1776ms]
range_agg Memory: 25.4 KB (-4.09%) Avg: 3.3695ms (+5.26%) Median: 3.3528ms (+5.19%) [3.3041ms .. 3.5736ms]
range_agg_with_avg_sub_agg Memory: 96.6 KB (+83.89%) Avg: 6.9758ms (-3.45%) Median: 6.9459ms (-3.93%) [6.8786ms .. 7.2956ms]
range_agg_with_term_agg_status Memory: 110.2 KB (+63.78%) Avg: 6.3969ms (-68.61%) Median: 6.3650ms (-68.90%) [6.3170ms .. 6.6804ms]
range_agg_with_term_agg_many Memory: 6.9 MB (+0.33%) Avg: 13.8903ms (-54.00%) Median: 13.8232ms (-53.74%) [13.4845ms .. 15.1131ms]
histogram Memory: 22.1 KB (-1.79%) Avg: 3.0648ms (-4.26%) Median: 3.0369ms (-4.20%) [3.0005ms .. 3.2348ms]
histogram_hard_bounds Memory: 20.9 KB (+4.30%) Avg: 1.5927ms (-1.19%) Median: 1.5824ms (-0.75%) [1.5506ms .. 1.7170ms]
histogram_with_avg_sub_agg Memory: 123.8 KB (+69.81%) Avg: 9.4157ms (+4.98%) Median: 9.3585ms (+5.21%) [9.2614ms .. 9.8650ms]
histogram_with_term_agg_status Memory: 494.1 KB (+5.11%) Avg: 12.7594ms (-34.76%) Median: 12.7048ms (-34.67%) [12.5948ms .. 13.2293ms]
avg_and_range_with_avg_sub_agg Memory: 84.2 KB (+100.85%) Avg: 9.8299ms (+2.59%) Median: 9.7713ms (+2.16%) [9.6709ms .. 10.2506ms]
filter_agg_all_query_count_agg Memory: 139.9 KB (+22.44%) Avg: 4.4621ms (+15.01%) Median: 4.4214ms (+14.57%) [4.3417ms .. 5.3741ms]
filter_agg_term_query_count_agg Memory: 140.3 KB (+22.11%) Avg: 6.8583ms (+9.89%) Median: 6.8357ms (+9.58%) [6.6912ms .. 7.1673ms]
filter_agg_all_query_with_sub_aggs Memory: 190.7 KB (+44.74%) Avg: 9.4598ms (+5.58%) Median: 9.3875ms (+5.69%) [9.2740ms .. 10.3321ms]
filter_agg_term_query_with_sub_aggs Memory: 191.0 KB (+44.62%) Avg: 11.8665ms (+5.17%) Median: 11.8311ms (+5.10%) [11.6456ms .. 12.3816ms]
