[WIP] feat: add dask windows
Previously dask supported "noop" windows. This PR adds some limited window functionality. Draft to see how far I can reasonably get while keeping this a reasonable size.
Notes:
- dask only supports single value sorts, so any window with grouping and sorting is not supported
- I'm trying to keep the structure quite close the pandas backend, which will make some eventual consolidation and refactoring much easier to do. However, the pandas windowing code definitely needs some broad refactoring so things look a bit messy.
Codecov Report
Merging #4157 (0ed1e89) into master (0393891) will increase coverage by
0.00%. The diff coverage is93.75%.
@@ Coverage Diff @@
## master #4157 +/- ##
=======================================
Coverage 92.98% 92.98%
=======================================
Files 158 158
Lines 18298 18373 +75
Branches 2663 2675 +12
=======================================
+ Hits 17014 17085 +71
- Misses 963 965 +2
- Partials 321 323 +2
| Flag | Coverage Δ | |
|---|---|---|
| Linux | 92.88% <93.75%> (+<0.01%) |
:arrow_up: |
| Windows | 72.13% <93.75%> (+0.10%) |
:arrow_up: |
| backend | 89.89% <93.75%> (+0.01%) |
:arrow_up: |
| clickhouse | 49.91% <0.00%> (-0.21%) |
:arrow_down: |
| core | 52.42% <0.00%> (-0.09%) |
:arrow_down: |
| dask | 52.23% <93.75%> (+0.41%) |
:arrow_up: |
| datafusion | 43.89% <0.00%> (-0.18%) |
:arrow_down: |
| duckdb | 49.67% <0.00%> (-0.21%) |
:arrow_down: |
| impala | 55.23% <0.00%> (-0.24%) |
:arrow_down: |
| mysql | 47.48% <0.00%> (-0.20%) |
:arrow_down: |
| pandas | 49.28% <0.00%> (-0.21%) |
:arrow_down: |
| postgres | 51.52% <0.00%> (-0.22%) |
:arrow_down: |
| pyspark | 52.75% <0.00%> (-0.22%) |
:arrow_down: |
| python-3.10.5 | 86.89% <93.75%> (+0.04%) |
:arrow_up: |
| python-3.8.10 | 72.13% <93.75%> (+0.10%) |
:arrow_up: |
| python-3.8.13 | 92.89% <93.75%> (+<0.01%) |
:arrow_up: |
| python-3.9.13 | 52.28% <0.00%> (-9.29%) |
:arrow_down: |
| sqlite | 48.95% <0.00%> (-0.21%) |
:arrow_down: |
| tpc | 35.17% <0.00%> (-0.15%) |
:arrow_down: |
| tpch | 35.17% <0.00%> (-0.15%) |
:arrow_down: |
Flags with carried forward coverage won't be shown. Click here to find out more.
| Impacted Files | Coverage Δ | |
|---|---|---|
| ibis/backends/dask/execution/util.py | 89.62% <83.33%> (+1.56%) |
:arrow_up: |
| ibis/backends/dask/execution/window.py | 95.68% <94.38%> (-4.32%) |
:arrow_down: |
| ibis/backends/dask/execution/selection.py | 87.23% <100.00%> (ø) |
|
| ibis/backends/impala/client.py | 87.17% <0.00%> (-0.66%) |
:arrow_down: |
| ibis/backends/dask/execution/arrays.py | 100.00% <0.00%> (+4.76%) |
:arrow_up: |
Unit Test Results
35 files 35 suites 1h 7m 54s :stopwatch: 8 662 tests 6 760 :heavy_check_mark: 1 665 :zzz: 0 :x: 237 :fire: 31 827 runs 25 235 :heavy_check_mark: 6 355 :zzz: 0 :x: 237 :fire:
For more details on these errors, see this check.
Results for commit 0ed1e894.
:recycle: This comment has been updated with latest results.
@gerrymanoim We're planning on a 3.2.0 release around the beginning of September. Do you think you'll be able to finish this up by then?
@gerrymanoim Are you still working on this?
Hey - apologies I missed your first message.
Yes - this is still on our list of things we want to do - I don't have an ETA unfortunately for when I'll next carve out time to work on this. You can close this if you'd like.
Closing. Please reopen if you get a chance to work on it later!