datafusion
datafusion copied to clipboard
Use arrow row format in SortPreservingMerge ~50-70% faster
Draft as depends on https://github.com/apache/arrow-rs/pull/2593
Which issue does this PR close?
Part of #416
Rationale for this change
merge i64 time: [18.361 ms 18.383 ms 18.406 ms]
change: [-53.779% -53.520% -53.283%] (p = 0.00 < 0.05)
Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) low mild
1 (1.00%) high mild
1 (1.00%) high severe
merge f64 time: [18.271 ms 18.289 ms 18.307 ms]
change: [-53.881% -53.730% -53.598%] (p = 0.00 < 0.05)
Performance has improved.
merge utf8 low cardinality
time: [17.168 ms 17.185 ms 17.203 ms]
change: [-62.941% -62.831% -62.731%] (p = 0.00 < 0.05)
Performance has improved.
merge utf8 high cardinality
time: [19.513 ms 19.539 ms 19.566 ms]
change: [-54.113% -54.022% -53.932%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
merge utf8 tuple time: [27.579 ms 27.608 ms 27.639 ms]
change: [-56.213% -56.134% -56.054%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
merge utf8 dictionary time: [16.251 ms 16.265 ms 16.280 ms]
change: [-65.210% -65.063% -64.945%] (p = 0.00 < 0.05)
Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild
merge utf8 dictionary tuple
time: [19.057 ms 19.081 ms 19.111 ms]
change: [-70.756% -70.581% -70.418%] (p = 0.00 < 0.05)
Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
2 (2.00%) low mild
1 (1.00%) high mild
1 (1.00%) high severe
merge mixed utf8 dictionary tuple
time: [24.586 ms 24.634 ms 24.684 ms]
change: [-63.732% -63.583% -63.439%] (p = 0.00 < 0.05)
Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) high mild
1 (1.00%) high severe
merge mixed tuple time: [27.034 ms 27.075 ms 27.119 ms]
change: [-47.178% -46.969% -46.762%] (p = 0.00 < 0.05)
Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) high mild
1 (1.00%) high severe
It is also worth highlighting that these benchmarks are in many ways the worst case, as the rows are distributed randomly across streams, instead of large contiguous slices, which increases the cost of reassembly, i.e. the non-comparison portion of the operator.