ibis icon indicating copy to clipboard operation
ibis copied to clipboard

[WIP] feat: add dask windows

Open gerrymanoim opened this issue 3 years ago • 3 comments

Previously dask supported "noop" windows. This PR adds some limited window functionality. Draft to see how far I can reasonably get while keeping this a reasonable size.

Notes:

  • dask only supports single value sorts, so any window with grouping and sorting is not supported
  • I'm trying to keep the structure quite close the pandas backend, which will make some eventual consolidation and refactoring much easier to do. However, the pandas windowing code definitely needs some broad refactoring so things look a bit messy.

gerrymanoim avatar Jun 29 '22 04:06 gerrymanoim

Codecov Report

Merging #4157 (0ed1e89) into master (0393891) will increase coverage by 0.00%. The diff coverage is 93.75%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #4157   +/-   ##
=======================================
  Coverage   92.98%   92.98%           
=======================================
  Files         158      158           
  Lines       18298    18373   +75     
  Branches     2663     2675   +12     
=======================================
+ Hits        17014    17085   +71     
- Misses        963      965    +2     
- Partials      321      323    +2     
Flag Coverage Δ
Linux 92.88% <93.75%> (+<0.01%) :arrow_up:
Windows 72.13% <93.75%> (+0.10%) :arrow_up:
backend 89.89% <93.75%> (+0.01%) :arrow_up:
clickhouse 49.91% <0.00%> (-0.21%) :arrow_down:
core 52.42% <0.00%> (-0.09%) :arrow_down:
dask 52.23% <93.75%> (+0.41%) :arrow_up:
datafusion 43.89% <0.00%> (-0.18%) :arrow_down:
duckdb 49.67% <0.00%> (-0.21%) :arrow_down:
impala 55.23% <0.00%> (-0.24%) :arrow_down:
mysql 47.48% <0.00%> (-0.20%) :arrow_down:
pandas 49.28% <0.00%> (-0.21%) :arrow_down:
postgres 51.52% <0.00%> (-0.22%) :arrow_down:
pyspark 52.75% <0.00%> (-0.22%) :arrow_down:
python-3.10.5 86.89% <93.75%> (+0.04%) :arrow_up:
python-3.8.10 72.13% <93.75%> (+0.10%) :arrow_up:
python-3.8.13 92.89% <93.75%> (+<0.01%) :arrow_up:
python-3.9.13 52.28% <0.00%> (-9.29%) :arrow_down:
sqlite 48.95% <0.00%> (-0.21%) :arrow_down:
tpc 35.17% <0.00%> (-0.15%) :arrow_down:
tpch 35.17% <0.00%> (-0.15%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
ibis/backends/dask/execution/util.py 89.62% <83.33%> (+1.56%) :arrow_up:
ibis/backends/dask/execution/window.py 95.68% <94.38%> (-4.32%) :arrow_down:
ibis/backends/dask/execution/selection.py 87.23% <100.00%> (ø)
ibis/backends/impala/client.py 87.17% <0.00%> (-0.66%) :arrow_down:
ibis/backends/dask/execution/arrays.py 100.00% <0.00%> (+4.76%) :arrow_up:

codecov[bot] avatar Jun 29 '22 04:06 codecov[bot]

Unit Test Results

       35 files         35 suites   1h 7m 54s :stopwatch:   8 662 tests   6 760 :heavy_check_mark: 1 665 :zzz: 0 :x: 237 :fire: 31 827 runs  25 235 :heavy_check_mark: 6 355 :zzz: 0 :x: 237 :fire:

For more details on these errors, see this check.

Results for commit 0ed1e894.

:recycle: This comment has been updated with latest results.

github-actions[bot] avatar Jun 29 '22 04:06 github-actions[bot]

@gerrymanoim We're planning on a 3.2.0 release around the beginning of September. Do you think you'll be able to finish this up by then?

cpcloud avatar Aug 16 '22 11:08 cpcloud

@gerrymanoim Are you still working on this?

cpcloud avatar Oct 06 '22 20:10 cpcloud

Hey - apologies I missed your first message.

Yes - this is still on our list of things we want to do - I don't have an ETA unfortunately for when I'll next carve out time to work on this. You can close this if you'd like.

gerrymanoim avatar Oct 07 '22 15:10 gerrymanoim

Closing. Please reopen if you get a chance to work on it later!

cpcloud avatar Oct 31 '22 20:10 cpcloud