databend icon indicating copy to clipboard operation
databend copied to clipboard

feat: implement partial full outer join for merge into I/O improvement

Open JackTan25 opened this issue 1 year ago โ€ข 0 comments

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

PartitialFullOuter is an optimization for merge into source build, in this case, we can do pass the target block of matched partition and ummatched partition, and append block directly, but it's limitted, we just support the matched partial block must be modfied(update or delete), so we need to make sure the last condition for matched expression is none. for example:

a. merge into t1 using t2 on t1.a = t2.a when matched and cond1 then xx when matched then xxx.
b. merge into t1 using t2 on t1.a = t2.a when matched and cond1 then xx when matched and cond2 then xxx.

we can support a not b. The reason for this limitation is that if a target block is not modfied, we will append the unmacthed partial of target block unexpectedly. image we use this optimization to improve merge into I/O.

Tests

  • [ ] Unit Test
  • [x] Logic Test
  • [ ] Benchmark Test
  • [ ] No Test - Explain why

Type of change

  • [ ] Bug Fix (non-breaking change which fixes an issue)
  • [ ] New Feature (non-breaking change which adds functionality)
  • [ ] Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • [ ] Documentation Update
  • [ ] Refactoring
  • [x] Performance Improvement
  • [ ] Other (please describe):

This change isโ€‚Reviewable

JackTan25 avatar Feb 21 '24 08:02 JackTan25