datafusion icon indicating copy to clipboard operation
datafusion copied to clipboard

Use arrow row format in SortPreservingMerge ~50-70% faster

Open tustvold opened this issue 3 years ago • 0 comments

Draft as depends on https://github.com/apache/arrow-rs/pull/2593

Which issue does this PR close?

Part of #416

Rationale for this change

merge i64               time:   [18.361 ms 18.383 ms 18.406 ms]                      
                        change: [-53.779% -53.520% -53.283%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild
  1 (1.00%) high severe

merge f64               time:   [18.271 ms 18.289 ms 18.307 ms]                      
                        change: [-53.881% -53.730% -53.598%] (p = 0.00 < 0.05)
                        Performance has improved.

merge utf8 low cardinality                                                                            
                        time:   [17.168 ms 17.185 ms 17.203 ms]
                        change: [-62.941% -62.831% -62.731%] (p = 0.00 < 0.05)
                        Performance has improved.

merge utf8 high cardinality                                                                            
                        time:   [19.513 ms 19.539 ms 19.566 ms]
                        change: [-54.113% -54.022% -53.932%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

merge utf8 tuple        time:   [27.579 ms 27.608 ms 27.639 ms]                             
                        change: [-56.213% -56.134% -56.054%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

merge utf8 dictionary   time:   [16.251 ms 16.265 ms 16.280 ms]                                  
                        change: [-65.210% -65.063% -64.945%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild

merge utf8 dictionary tuple                                                                            
                        time:   [19.057 ms 19.081 ms 19.111 ms]
                        change: [-70.756% -70.581% -70.418%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) low mild
  1 (1.00%) high mild
  1 (1.00%) high severe

merge mixed utf8 dictionary tuple                                                                            
                        time:   [24.586 ms 24.634 ms 24.684 ms]
                        change: [-63.732% -63.583% -63.439%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

merge mixed tuple       time:   [27.034 ms 27.075 ms 27.119 ms]                              
                        change: [-47.178% -46.969% -46.762%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

It is also worth highlighting that these benchmarks are in many ways the worst case, as the rows are distributed randomly across streams, instead of large contiguous slices, which increases the cost of reassembly, i.e. the non-comparison portion of the operator.

What changes are included in this PR?

Are there any user-facing changes?

tustvold avatar Sep 07 '22 12:09 tustvold