databend icon indicating copy to clipboard operation
databend copied to clipboard

Tracking: enhance hash join spilling

Open xudong963 opened this issue 1 year ago • 5 comments

Summary

  • [x] https://github.com/datafuselabs/databend/issues/14771
  • [x] make partition number configurable: part of https://github.com/datafuselabs/databend/pull/14794
  • [x] support left-related join spilling: #14828
  • [x] make hash join spilling stable and add enough tests: part of https://github.com/datafuselabs/databend/pull/14794
  • [x] support right-related join spilling: https://github.com/datafuselabs/databend/pull/14853
  • [x] open hash join spilling by default: https://github.com/datafuselabs/databend/pull/14843
  • [x] support cross join spilling: https://github.com/datafuselabs/databend/pull/14887
  • [x] https://github.com/datafuselabs/databend/issues/14963
  • [ ] support recursive spilling when the restored partition still can't fit into memory.

xudong963 avatar Feb 26 '24 02:02 xudong963

Bug fixes:

  • https://github.com/datafuselabs/databend/pull/14733
  • https://github.com/datafuselabs/databend/pull/14794
  • https://github.com/datafuselabs/databend/pull/14860
  • https://github.com/datafuselabs/databend/pull/15089

xudong963 avatar Feb 26 '24 05:02 xudong963

Refactor:

  • https://github.com/datafuselabs/databend/pull/14743
  • https://github.com/datafuselabs/databend/pull/14768
  • https://github.com/datafuselabs/databend/pull/14919
  • https://github.com/datafuselabs/databend/pull/14942

xudong963 avatar Feb 26 '24 13:02 xudong963

Others:

  • A repo to test databend's join spilling: https://github.com/xudong963/NOOOM

xudong963 avatar Mar 12 '24 09:03 xudong963

Others:

  • A repo to test databend's join spilling: https://github.com/xudong963/NOOOM

Great, I didn't know have this one. Can you port this into wizard? https://github.com/datafuselabs/wizard

bohutang avatar Mar 13 '24 00:03 bohutang

Others:

  • A repo to test databend's join spilling: https://github.com/xudong963/NOOOM

Great, I didn't know have this one. Can you port this into wizard? https://github.com/datafuselabs/wizard

Will do it

xudong963 avatar Mar 13 '24 08:03 xudong963