paimon icon indicating copy to clipboard operation
paimon copied to clipboard

[core] Skip processed sequence group fields to improve performance of PartialUpdateMergeFunction

Open liyubin117 opened this issue 3 months ago • 3 comments

Purpose

Skip processed sequence group fields to improve performance of PartialUpdateMergeFunction

  1. skip null sequence group fields
  2. skip sequence group fields have been updated at once
  3. move seqComparator != null check forward for common case optimization

Tests

  1. PartialUpdateMergeFunctionTest
  2. PartialUpdateMergeFunctionBenchmark new
Running benchmark: partial-update-benchmark
  Running case: updateWithSequenceGroup
Iteration 0 took 13835399 microseconds
Iteration 1 took 13355838 microseconds
Iteration 2 took 13048940 microseconds
Iteration 3 took 13016992 microseconds
Iteration 4 took 13141102 microseconds
  Stopped after 5 iterations, 66398 ms

OpenJDK 64-Bit Server VM 1.8.0_412-b08 on Mac OS X 14.4.1
Apple M1 Pro
partial-update-benchmark:                                                                            Best/Avg Time(ms)    Row Rate(K/s)      Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
OPERATORTEST_partial-update-benchmark_updateWithSequenceGroup                                           13017 / 13280           3072.9            325.4       1.0X

Running benchmark: partial-update-benchmark
  Running case: retractWithSequenceGroup
Iteration 0 took 12762749 microseconds
Iteration 1 took 12572677 microseconds
Iteration 2 took 12441134 microseconds
Iteration 3 took 12490136 microseconds
Iteration 4 took 12677572 microseconds
  Stopped after 5 iterations, 62944 ms

OpenJDK 64-Bit Server VM 1.8.0_412-b08 on Mac OS X 14.4.1
Apple M1 Pro
partial-update-benchmark:                                                                            Best/Avg Time(ms)    Row Rate(K/s)      Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
OPERATORTEST_partial-update-benchmark_retractWithSequenceGroup                                          12441 / 12589           3215.1            311.0       1.0X

Running benchmark: partial-update-benchmark
  Running case: updateWithEmptySequenceGroup
Iteration 0 took 10842099 microseconds
Iteration 1 took 10364416 microseconds
Iteration 2 took 10588347 microseconds
Iteration 3 took 10492397 microseconds
Iteration 4 took 10618167 microseconds
  Stopped after 5 iterations, 52905 ms

OpenJDK 64-Bit Server VM 1.8.0_412-b08 on Mac OS X 14.4.1
Apple M1 Pro
partial-update-benchmark:                                                                            Best/Avg Time(ms)    Row Rate(K/s)      Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
OPERATORTEST_partial-update-benchmark_updateWithEmptySequenceGroup                                      10364 / 10581           3859.4            259.1       1.0X

old

Running benchmark: partial-update-benchmark
  Running case: updateWithSequenceGroup
Iteration 0 took 14191378 microseconds
Iteration 1 took 14021648 microseconds
Iteration 2 took 14039543 microseconds
Iteration 3 took 13882215 microseconds
Iteration 4 took 13819146 microseconds
  Stopped after 5 iterations, 69953 ms

OpenJDK 64-Bit Server VM 1.8.0_412-b08 on Mac OS X 14.4.1
Apple M1 Pro
partial-update-benchmark:                                                                            Best/Avg Time(ms)    Row Rate(K/s)      Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
OPERATORTEST_partial-update-benchmark_updateWithSequenceGroup                                           13819 / 13991           2894.5            345.5       1.0X

Running benchmark: partial-update-benchmark
  Running case: retractWithSequenceGroup
Iteration 0 took 14462734 microseconds
Iteration 1 took 14674714 microseconds
Iteration 2 took 15522533 microseconds
Iteration 3 took 15111800 microseconds
Iteration 4 took 14651207 microseconds
  Stopped after 5 iterations, 74422 ms

OpenJDK 64-Bit Server VM 1.8.0_412-b08 on Mac OS X 14.4.1
Apple M1 Pro
partial-update-benchmark:                                                                            Best/Avg Time(ms)    Row Rate(K/s)      Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
OPERATORTEST_partial-update-benchmark_retractWithSequenceGroup                                          14463 / 14885           2765.7            361.6       1.0X

Running benchmark: partial-update-benchmark
  Running case: updateWithEmptySequenceGroup
Iteration 0 took 10743736 microseconds
Iteration 1 took 11192192 microseconds
Iteration 2 took 12587153 microseconds
Iteration 3 took 11009585 microseconds
Iteration 4 took 10661533 microseconds
  Stopped after 5 iterations, 56194 ms

OpenJDK 64-Bit Server VM 1.8.0_412-b08 on Mac OS X 14.4.1
Apple M1 Pro
partial-update-benchmark:                                                                            Best/Avg Time(ms)    Row Rate(K/s)      Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
OPERATORTEST_partial-update-benchmark_updateWithEmptySequenceGroup                                      10662 / 11239           3751.8            266.5       1.0X

liyubin117 avatar Sep 28 '25 03:09 liyubin117

@JingsongLi All CI succeeded extra non-related hive metastore issue, PTAL, thanks :) CI succeeded in my repo. image

liyubin117 avatar Sep 28 '25 07:09 liyubin117

@JingsongLi CI succeeded

liyubin117 avatar Sep 29 '25 03:09 liyubin117

@yuzelin Hi, PTAL, thanks:)

liyubin117 avatar Oct 29 '25 12:10 liyubin117