datafusion
datafusion copied to clipboard
apache
Draft: Use upstream arrow `coalesce` kernel in DataFusion
Which issue does this PR close?
- Related to https://github.com/apache/arrow-rs/issues/6692
- Related to #3463
Rationale for this change
I am trying to move the coalesce operation upstream into arrow-rs so that it can be reused in parquet filter pushdown and more highly optimized. See :
- https://github.com/apache/arrow-rs/pull/7597
The proposed upstream coalescer API is slightly different (guaranteed batch size, and doesn't have limit) so we must adapt the DataFusion code to handle this
What changes are included in this PR?
This PR refactors the BatchCoalescer in DataFusion to use the proposed upstream API to show that it
- Can be used (api is complete enough)
- Is not any slower
Are these changes tested?
By CI tests
Are there any user-facing changes?
No
🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.11.0-1013-gcp #13~24.04.1-Ubuntu SMP Wed Apr 2 16:34:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/test_upstream_coalesce (e79454f48929360b2753c51d9647f9a302c9e184) to 992d156c46f6ad4f0096c4a62b293cabef63718d diff
Benchmarks: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
🤖: Benchmark completed
Details
Comparing HEAD and alamb_test_upstream_coalesce
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_test_upstream_coalesce ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0 │ 1903.28ms │ 1910.47ms │ no change │
│ QQuery 1 │ 705.78ms │ 705.39ms │ no change │
│ QQuery 2 │ 1416.79ms │ 1427.68ms │ no change │
│ QQuery 3 │ 721.59ms │ 695.62ms │ no change │
│ QQuery 4 │ 1438.61ms │ 1459.22ms │ no change │
│ QQuery 5 │ 15147.47ms │ 15620.61ms │ no change │
│ QQuery 6 │ 2025.79ms │ 2037.42ms │ no change │
│ QQuery 7 │ 2088.81ms │ 2069.21ms │ no change │
│ QQuery 8 │ 852.36ms │ 834.76ms │ no change │
└──────────────┴────────────┴──────────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 26300.48ms │
│ Total Time (alamb_test_upstream_coalesce) │ 26760.37ms │
│ Average Time (HEAD) │ 2922.28ms │
│ Average Time (alamb_test_upstream_coalesce) │ 2973.37ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 9 │
└─────────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_test_upstream_coalesce ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 15.39ms │ 16.12ms │ no change │
│ QQuery 1 │ 33.30ms │ 33.27ms │ no change │
│ QQuery 2 │ 81.95ms │ 81.53ms │ no change │
│ QQuery 3 │ 99.01ms │ 95.19ms │ no change │
│ QQuery 4 │ 680.72ms │ 583.09ms │ +1.17x faster │
│ QQuery 5 │ 901.62ms │ 859.72ms │ no change │
│ QQuery 6 │ 24.40ms │ 23.86ms │ no change │
│ QQuery 7 │ 37.45ms │ 36.23ms │ no change │
│ QQuery 8 │ 951.27ms │ 922.01ms │ no change │
│ QQuery 9 │ 1259.35ms │ 1187.14ms │ +1.06x faster │
│ QQuery 10 │ 270.07ms │ 260.94ms │ no change │
│ QQuery 11 │ 301.01ms │ 298.82ms │ no change │
│ QQuery 12 │ 929.60ms │ 920.78ms │ no change │
│ QQuery 13 │ 1352.62ms │ 1355.67ms │ no change │
│ QQuery 14 │ 843.74ms │ 858.27ms │ no change │
│ QQuery 15 │ 842.01ms │ 841.71ms │ no change │
│ QQuery 16 │ 1725.31ms │ 1755.26ms │ no change │
│ QQuery 17 │ 1605.81ms │ 1607.33ms │ no change │
│ QQuery 18 │ 3061.08ms │ 3122.06ms │ no change │
│ QQuery 19 │ 84.26ms │ 81.68ms │ no change │
│ QQuery 20 │ 1134.75ms │ 1120.94ms │ no change │
│ QQuery 21 │ 1327.65ms │ 1341.47ms │ no change │
│ QQuery 22 │ 2173.10ms │ 2220.13ms │ no change │
│ QQuery 23 │ 8004.42ms │ 7989.85ms │ no change │
│ QQuery 24 │ 468.53ms │ 466.85ms │ no change │
│ QQuery 25 │ 383.28ms │ 389.97ms │ no change │
│ QQuery 26 │ 538.11ms │ 527.43ms │ no change │
│ QQuery 27 │ 1606.15ms │ 1622.52ms │ no change │
│ QQuery 28 │ 13703.82ms │ 13070.74ms │ no change │
│ QQuery 29 │ 520.31ms │ 525.56ms │ no change │
│ QQuery 30 │ 796.51ms │ 839.14ms │ 1.05x slower │
│ QQuery 31 │ 851.63ms │ 884.19ms │ no change │
│ QQuery 32 │ 2618.12ms │ 2636.26ms │ no change │
│ QQuery 33 │ 3372.88ms │ 3288.44ms │ no change │
│ QQuery 34 │ 3403.04ms │ 3349.20ms │ no change │
│ QQuery 35 │ 1296.66ms │ 1313.52ms │ no change │
│ QQuery 36 │ 121.58ms │ 126.46ms │ no change │
│ QQuery 37 │ 56.05ms │ 56.07ms │ no change │
│ QQuery 38 │ 122.31ms │ 124.83ms │ no change │
│ QQuery 39 │ 197.01ms │ 198.32ms │ no change │
│ QQuery 40 │ 50.25ms │ 47.22ms │ +1.06x faster │
│ QQuery 41 │ 43.47ms │ 45.99ms │ 1.06x slower │
│ QQuery 42 │ 37.67ms │ 38.74ms │ no change │
└──────────────┴────────────┴──────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 57927.27ms │
│ Total Time (alamb_test_upstream_coalesce) │ 57164.53ms │
│ Average Time (HEAD) │ 1347.15ms │
│ Average Time (alamb_test_upstream_coalesce) │ 1329.41ms │
│ Queries Faster │ 3 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 38 │
└─────────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_test_upstream_coalesce ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1 │ 116.93ms │ 116.77ms │ no change │
│ QQuery 2 │ 22.28ms │ 22.65ms │ no change │
│ QQuery 3 │ 34.38ms │ 35.42ms │ no change │
│ QQuery 4 │ 19.66ms │ 20.79ms │ 1.06x slower │
│ QQuery 5 │ 54.59ms │ 53.23ms │ no change │
│ QQuery 6 │ 11.97ms │ 11.95ms │ no change │
│ QQuery 7 │ 96.84ms │ 100.06ms │ no change │
│ QQuery 8 │ 25.69ms │ 25.35ms │ no change │
│ QQuery 9 │ 60.26ms │ 60.42ms │ no change │
│ QQuery 10 │ 49.39ms │ 47.08ms │ no change │
│ QQuery 11 │ 11.41ms │ 12.24ms │ 1.07x slower │
│ QQuery 12 │ 41.31ms │ 40.07ms │ no change │
│ QQuery 13 │ 27.65ms │ 28.44ms │ no change │
│ QQuery 14 │ 9.80ms │ 9.94ms │ no change │
│ QQuery 15 │ 23.05ms │ 22.78ms │ no change │
│ QQuery 16 │ 21.49ms │ 20.98ms │ no change │
│ QQuery 17 │ 98.87ms │ 98.30ms │ no change │
│ QQuery 18 │ 217.87ms │ 208.15ms │ no change │
│ QQuery 19 │ 26.67ms │ 26.21ms │ no change │
│ QQuery 20 │ 34.27ms │ 36.07ms │ 1.05x slower │
│ QQuery 21 │ 160.60ms │ 163.49ms │ no change │
│ QQuery 22 │ 16.66ms │ 16.30ms │ no change │
└──────────────┴──────────┴──────────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1181.64ms │
│ Total Time (alamb_test_upstream_coalesce) │ 1176.69ms │
│ Average Time (HEAD) │ 53.71ms │
│ Average Time (alamb_test_upstream_coalesce) │ 53.49ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 3 │
│ Queries with No Change │ 19 │
└─────────────────────────────────────────────┴───────────┘
🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.11.0-1013-gcp #13~24.04.1-Ubuntu SMP Wed Apr 2 16:34:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/test_upstream_coalesce (9e20973387ae4b04914917726c834120e592238d) to 992d156c46f6ad4f0096c4a62b293cabef63718d diff
Benchmarks: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
🤖: Benchmark completed
Details
Comparing HEAD and alamb_test_upstream_coalesce
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_test_upstream_coalesce ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0 │ 1914.38ms │ 1938.51ms │ no change │
│ QQuery 1 │ 711.81ms │ 718.29ms │ no change │
│ QQuery 2 │ 1455.52ms │ 1470.28ms │ no change │
│ QQuery 3 │ 712.37ms │ 694.68ms │ no change │
│ QQuery 4 │ 1449.70ms │ 1448.88ms │ no change │
│ QQuery 5 │ 15655.93ms │ 15613.27ms │ no change │
│ QQuery 6 │ 2014.71ms │ 2072.03ms │ no change │
│ QQuery 7 │ 2080.97ms │ 2169.52ms │ no change │
│ QQuery 8 │ 856.77ms │ 841.42ms │ no change │
└──────────────┴────────────┴──────────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 26852.14ms │
│ Total Time (alamb_test_upstream_coalesce) │ 26966.88ms │
│ Average Time (HEAD) │ 2983.57ms │
│ Average Time (alamb_test_upstream_coalesce) │ 2996.32ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 9 │
└─────────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_test_upstream_coalesce ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 15.50ms │ 15.07ms │ no change │
│ QQuery 1 │ 32.04ms │ 32.46ms │ no change │
│ QQuery 2 │ 85.71ms │ 81.49ms │ no change │
│ QQuery 3 │ 101.37ms │ 96.91ms │ no change │
│ QQuery 4 │ 635.37ms │ 605.83ms │ no change │
│ QQuery 5 │ 846.37ms │ 857.35ms │ no change │
│ QQuery 6 │ 25.51ms │ 22.94ms │ +1.11x faster │
│ QQuery 7 │ 36.44ms │ 36.85ms │ no change │
│ QQuery 8 │ 905.00ms │ 899.47ms │ no change │
│ QQuery 9 │ 1221.22ms │ 1203.33ms │ no change │
│ QQuery 10 │ 256.84ms │ 261.35ms │ no change │
│ QQuery 11 │ 298.30ms │ 293.02ms │ no change │
│ QQuery 12 │ 896.38ms │ 900.13ms │ no change │
│ QQuery 13 │ 1319.11ms │ 1229.16ms │ +1.07x faster │
│ QQuery 14 │ 821.82ms │ 840.52ms │ no change │
│ QQuery 15 │ 808.55ms │ 832.04ms │ no change │
│ QQuery 16 │ 1708.26ms │ 1717.94ms │ no change │
│ QQuery 17 │ 1564.42ms │ 1600.58ms │ no change │
│ QQuery 18 │ 3036.53ms │ 3090.72ms │ no change │
│ QQuery 19 │ 83.52ms │ 83.60ms │ no change │
│ QQuery 20 │ 1120.03ms │ 1142.33ms │ no change │
│ QQuery 21 │ 1327.78ms │ 1340.28ms │ no change │
│ QQuery 22 │ 2180.54ms │ 2211.59ms │ no change │
│ QQuery 23 │ 8051.83ms │ 8092.34ms │ no change │
│ QQuery 24 │ 467.15ms │ 463.58ms │ no change │
│ QQuery 25 │ 393.84ms │ 390.05ms │ no change │
│ QQuery 26 │ 540.91ms │ 522.58ms │ no change │
│ QQuery 27 │ 1583.35ms │ 1648.93ms │ no change │
│ QQuery 28 │ 13535.80ms │ 14249.61ms │ 1.05x slower │
│ QQuery 29 │ 517.83ms │ 530.48ms │ no change │
│ QQuery 30 │ 794.70ms │ 826.55ms │ no change │
│ QQuery 31 │ 845.08ms │ 885.81ms │ no change │
│ QQuery 32 │ 2690.84ms │ 2696.00ms │ no change │
│ QQuery 33 │ 3326.15ms │ 3318.91ms │ no change │
│ QQuery 34 │ 3395.28ms │ 3349.97ms │ no change │
│ QQuery 35 │ 1271.25ms │ 1254.09ms │ no change │
│ QQuery 36 │ 126.96ms │ 133.46ms │ 1.05x slower │
│ QQuery 37 │ 55.58ms │ 55.47ms │ no change │
│ QQuery 38 │ 121.97ms │ 123.72ms │ no change │
│ QQuery 39 │ 197.25ms │ 197.97ms │ no change │
│ QQuery 40 │ 46.12ms │ 47.56ms │ no change │
│ QQuery 41 │ 45.89ms │ 45.86ms │ no change │
│ QQuery 42 │ 37.90ms │ 38.37ms │ no change │
└──────────────┴────────────┴──────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 57372.28ms │
│ Total Time (alamb_test_upstream_coalesce) │ 58266.22ms │
│ Average Time (HEAD) │ 1334.24ms │
│ Average Time (alamb_test_upstream_coalesce) │ 1355.03ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 39 │
└─────────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_test_upstream_coalesce ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 118.13ms │ 113.29ms │ no change │
│ QQuery 2 │ 22.88ms │ 21.65ms │ +1.06x faster │
│ QQuery 3 │ 34.71ms │ 34.51ms │ no change │
│ QQuery 4 │ 19.51ms │ 20.75ms │ 1.06x slower │
│ QQuery 5 │ 52.61ms │ 52.95ms │ no change │
│ QQuery 6 │ 11.99ms │ 12.01ms │ no change │
│ QQuery 7 │ 95.89ms │ 90.98ms │ +1.05x faster │
│ QQuery 8 │ 25.59ms │ 25.43ms │ no change │
│ QQuery 9 │ 59.49ms │ 57.39ms │ no change │
│ QQuery 10 │ 47.76ms │ 47.09ms │ no change │
│ QQuery 11 │ 11.30ms │ 11.17ms │ no change │
│ QQuery 12 │ 40.26ms │ 41.39ms │ no change │
│ QQuery 13 │ 26.70ms │ 27.66ms │ no change │
│ QQuery 14 │ 9.68ms │ 9.63ms │ no change │
│ QQuery 15 │ 22.07ms │ 22.57ms │ no change │
│ QQuery 16 │ 21.15ms │ 20.92ms │ no change │
│ QQuery 17 │ 95.43ms │ 96.63ms │ no change │
│ QQuery 18 │ 214.02ms │ 191.75ms │ +1.12x faster │
│ QQuery 19 │ 28.84ms │ 25.39ms │ +1.14x faster │
│ QQuery 20 │ 35.16ms │ 34.99ms │ no change │
│ QQuery 21 │ 157.53ms │ 161.74ms │ no change │
│ QQuery 22 │ 15.87ms │ 15.50ms │ no change │
└──────────────┴──────────┴──────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1166.59ms │
│ Total Time (alamb_test_upstream_coalesce) │ 1135.38ms │
│ Average Time (HEAD) │ 53.03ms │
│ Average Time (alamb_test_upstream_coalesce) │ 51.61ms │
│ Queries Faster │ 4 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 17 │
└─────────────────────────────────────────────┴───────────┘
🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.11.0-1013-gcp #13~24.04.1-Ubuntu SMP Wed Apr 2 16:34:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/test_upstream_coalesce (7d6471b5c9220d013cadd7313c655081af7115af) to 1daa5ed5cc51546904d45e23cc148601d973942a diff
Benchmarks: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
https://github.com/apache/datafusion/actions/runs/15506836203/job/43662848611?pr=16249
Caused by: process didn't exit successfully:
/home/runner/work/datafusion/datafusion/target/ci/deps/fuzz-a456cc2e1bdd6367(signal: 11, SIGSEGV: invalid memory reference) Error: Process completed with exit code 101.
🤔 that isn't good
I added a PR for reverting the changes in arrow-rs https://github.com/apache/arrow-rs/pull/7623 - probably something subtle with one of the fast paths that isn't tested in arrow-rs.
I added a PR for reverting the changes in arrow-rs apache/arrow-rs#7623 - probably something subtle with one of the fast paths that isn't tested in arrow-rs.
Thank you. I have a theory: https://github.com/apache/arrow-rs/pull/7623#discussion_r2133748478
🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.11.0-1013-gcp #13~24.04.1-Ubuntu SMP Wed Apr 2 16:34:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/test_upstream_coalesce (7d6471b5c9220d013cadd7313c655081af7115af) to 1daa5ed5cc51546904d45e23cc148601d973942a diff
Benchmarks: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.11.0-1013-gcp #13~24.04.1-Ubuntu SMP Wed Apr 2 16:34:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/test_upstream_coalesce (ea8b7000d7d804c532cd57b84bdf98c332904503) to 992d156c46f6ad4f0096c4a62b293cabef63718d diff
Benchmarks: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
🤖: Benchmark completed
Details
Comparing HEAD and alamb_test_upstream_coalesce
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_test_upstream_coalesce ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1889.06 ms │ 1880.96 ms │ no change │
│ QQuery 1 │ 716.77 ms │ 667.48 ms │ +1.07x faster │
│ QQuery 2 │ 1442.19 ms │ 1381.98 ms │ no change │
│ QQuery 3 │ 712.29 ms │ 707.44 ms │ no change │
│ QQuery 4 │ 1455.16 ms │ 1475.63 ms │ no change │
│ QQuery 5 │ 15533.57 ms │ 15474.07 ms │ no change │
│ QQuery 6 │ 2038.22 ms │ 2036.86 ms │ no change │
│ QQuery 7 │ 2061.78 ms │ 2037.80 ms │ no change │
│ QQuery 8 │ 842.37 ms │ 847.71 ms │ no change │
└──────────────┴─────────────┴──────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 26691.41ms │
│ Total Time (alamb_test_upstream_coalesce) │ 26509.93ms │
│ Average Time (HEAD) │ 2965.71ms │
│ Average Time (alamb_test_upstream_coalesce) │ 2945.55ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 8 │
│ Queries with Failure │ 0 │
└─────────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_test_upstream_coalesce ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 15.65 ms │ 15.15 ms │ no change │
│ QQuery 1 │ 32.30 ms │ 33.35 ms │ no change │
│ QQuery 2 │ 81.46 ms │ 81.55 ms │ no change │
│ QQuery 3 │ 96.00 ms │ 95.49 ms │ no change │
│ QQuery 4 │ 590.78 ms │ 601.11 ms │ no change │
│ QQuery 5 │ 854.48 ms │ 848.50 ms │ no change │
│ QQuery 6 │ 23.10 ms │ 22.76 ms │ no change │
│ QQuery 7 │ 37.06 ms │ 36.92 ms │ no change │
│ QQuery 8 │ 896.05 ms │ 900.08 ms │ no change │
│ QQuery 9 │ 1200.36 ms │ 1191.28 ms │ no change │
│ QQuery 10 │ 262.49 ms │ 251.45 ms │ no change │
│ QQuery 11 │ 291.65 ms │ 280.36 ms │ no change │
│ QQuery 12 │ 904.38 ms │ 889.86 ms │ no change │
│ QQuery 13 │ 1315.11 ms │ 1315.16 ms │ no change │
│ QQuery 14 │ 837.89 ms │ 834.22 ms │ no change │
│ QQuery 15 │ 821.30 ms │ 832.64 ms │ no change │
│ QQuery 16 │ 1705.68 ms │ 1693.59 ms │ no change │
│ QQuery 17 │ 1592.92 ms │ 1569.47 ms │ no change │
│ QQuery 18 │ 3045.50 ms │ 3002.20 ms │ no change │
│ QQuery 19 │ 84.78 ms │ 83.02 ms │ no change │
│ QQuery 20 │ 1120.03 ms │ 1133.34 ms │ no change │
│ QQuery 21 │ 1315.09 ms │ 1347.90 ms │ no change │
│ QQuery 22 │ 2178.63 ms │ 2211.92 ms │ no change │
│ QQuery 23 │ 8029.68 ms │ 8083.35 ms │ no change │
│ QQuery 24 │ 461.71 ms │ 449.41 ms │ no change │
│ QQuery 25 │ 382.66 ms │ 390.38 ms │ no change │
│ QQuery 26 │ 529.64 ms │ 522.68 ms │ no change │
│ QQuery 27 │ 1625.55 ms │ 1625.82 ms │ no change │
│ QQuery 28 │ 13784.35 ms │ 12932.17 ms │ +1.07x faster │
│ QQuery 29 │ 521.18 ms │ 524.09 ms │ no change │
│ QQuery 30 │ 801.66 ms │ 823.84 ms │ no change │
│ QQuery 31 │ 851.85 ms │ 852.68 ms │ no change │
│ QQuery 32 │ 2690.63 ms │ 2635.40 ms │ no change │
│ QQuery 33 │ 3345.68 ms │ 3295.22 ms │ no change │
│ QQuery 34 │ 3342.67 ms │ 3308.73 ms │ no change │
│ QQuery 35 │ 1263.11 ms │ 1267.54 ms │ no change │
│ QQuery 36 │ 123.69 ms │ 124.38 ms │ no change │
│ QQuery 37 │ 56.37 ms │ 58.61 ms │ no change │
│ QQuery 38 │ 125.62 ms │ 124.93 ms │ no change │
│ QQuery 39 │ 196.76 ms │ 196.00 ms │ no change │
│ QQuery 40 │ 46.66 ms │ 46.58 ms │ no change │
│ QQuery 41 │ 45.72 ms │ 46.90 ms │ no change │
│ QQuery 42 │ 38.86 ms │ 37.39 ms │ no change │
└──────────────┴─────────────┴──────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 57566.75ms │
│ Total Time (alamb_test_upstream_coalesce) │ 56617.44ms │
│ Average Time (HEAD) │ 1338.76ms │
│ Average Time (alamb_test_upstream_coalesce) │ 1316.68ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 42 │
│ Queries with Failure │ 0 │
└─────────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_test_upstream_coalesce ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 118.73 ms │ 102.68 ms │ +1.16x faster │
│ QQuery 2 │ 22.86 ms │ 21.30 ms │ +1.07x faster │
│ QQuery 3 │ 34.24 ms │ 33.92 ms │ no change │
│ QQuery 4 │ 20.26 ms │ 20.20 ms │ no change │
│ QQuery 5 │ 53.26 ms │ 50.18 ms │ +1.06x faster │
│ QQuery 6 │ 12.20 ms │ 11.80 ms │ no change │
│ QQuery 7 │ 98.01 ms │ 90.17 ms │ +1.09x faster │
│ QQuery 8 │ 26.39 ms │ 25.29 ms │ no change │
│ QQuery 9 │ 57.58 ms │ 57.16 ms │ no change │
│ QQuery 10 │ 49.82 ms │ 45.13 ms │ +1.10x faster │
│ QQuery 11 │ 11.63 ms │ 11.14 ms │ no change │
│ QQuery 12 │ 40.85 ms │ 38.18 ms │ +1.07x faster │
│ QQuery 13 │ 28.41 ms │ 28.02 ms │ no change │
│ QQuery 14 │ 10.05 ms │ 9.69 ms │ no change │
│ QQuery 15 │ 22.60 ms │ 22.41 ms │ no change │
│ QQuery 16 │ 21.50 ms │ 20.11 ms │ +1.07x faster │
│ QQuery 17 │ 97.15 ms │ 95.22 ms │ no change │
│ QQuery 18 │ 210.39 ms │ 176.64 ms │ +1.19x faster │
│ QQuery 19 │ 26.46 ms │ 25.16 ms │ no change │
│ QQuery 20 │ 34.93 ms │ 33.86 ms │ no change │
│ QQuery 21 │ 160.28 ms │ 144.59 ms │ +1.11x faster │
│ QQuery 22 │ 17.04 ms │ 15.31 ms │ +1.11x faster │
└──────────────┴───────────┴──────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1174.64ms │
│ Total Time (alamb_test_upstream_coalesce) │ 1078.14ms │
│ Average Time (HEAD) │ 53.39ms │
│ Average Time (alamb_test_upstream_coalesce) │ 49.01ms │
│ Queries Faster │ 10 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 12 │
│ Queries with Failure │ 0 │
└─────────────────────────────────────────────┴───────────┘
🤖: Benchmark completed
Details
This is similar to my results, some larger gains on TPC-H, smaller gains (spreaded out over different queries) for clickbench.
This is similar to my results, some larger gains on TPC-H, smaller gains (spreaded out over different queries) for clickbench.
SWEET!
Just wait until we get rid of the intermediate copy ;) it is going to be amazing
This is for me the result of not coalescing larger batches:
--- i/datafusion/physical-plan/src/coalesce/mod.rs
+++ w/datafusion/physical-plan/src/coalesce/mod.rs
@@ -33,6 +33,9 @@ pub struct LimitedBatchCoalescer {
fetch: Option<usize>,
/// Indicates if the coalescer is finished
finished: bool,
+
+ /// target batch size
+ target_batch_size: usize,
}
impl LimitedBatchCoalescer {
@@ -53,6 +56,7 @@ impl LimitedBatchCoalescer {
total_rows: 0,
fetch,
finished: false,
+ target_batch_size,
}
}
@@ -92,9 +96,15 @@ impl LimitedBatchCoalescer {
}
}
+ let num_rows = batch.num_rows();
+
self.total_rows += batch.num_rows();
self.inner.push_batch(batch)?;
+ if num_rows > self.target_batch_size / 2 {
+ self.inner.finish_buffered_batch()?;
+ }
+
Ok(false) // not at limit
}
Looks like some queries benefit from this:
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ main ┃ upstream_coalesce ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 62.59 ms │ 47.70 ms │ +1.31x faster │
│ QQuery 2 │ 12.52 ms │ 12.70 ms │ no change │
│ QQuery 3 │ 20.40 ms │ 20.52 ms │ no change │
│ QQuery 4 │ 13.52 ms │ 12.76 ms │ +1.06x faster │
│ QQuery 5 │ 31.40 ms │ 30.72 ms │ no change │
│ QQuery 6 │ 11.01 ms │ 11.13 ms │ no change │
│ QQuery 7 │ 67.99 ms │ 68.61 ms │ no change │
│ QQuery 8 │ 17.07 ms │ 16.47 ms │ no change │
│ QQuery 9 │ 38.69 ms │ 35.99 ms │ +1.08x faster │
│ QQuery 10 │ 32.05 ms │ 28.81 ms │ +1.11x faster │
│ QQuery 11 │ 5.80 ms │ 5.78 ms │ no change │
│ QQuery 12 │ 28.83 ms │ 27.93 ms │ no change │
│ QQuery 13 │ 18.29 ms │ 17.37 ms │ +1.05x faster │
│ QQuery 14 │ 5.24 ms │ 5.68 ms │ 1.08x slower │
│ QQuery 15 │ 10.88 ms │ 11.41 ms │ no change │
│ QQuery 16 │ 13.53 ms │ 12.77 ms │ +1.06x faster │
│ QQuery 17 │ 54.94 ms │ 62.53 ms │ 1.14x slower │
│ QQuery 18 │ 118.23 ms │ 115.08 ms │ no change │
│ QQuery 19 │ 21.28 ms │ 20.00 ms │ +1.06x faster │
│ QQuery 20 │ 20.43 ms │ 19.09 ms │ +1.07x faster │
│ QQuery 21 │ 84.41 ms │ 82.41 ms │ no change │
│ QQuery 22 │ 12.25 ms │ 10.74 ms │ +1.14x faster │
└──────────────┴───────────┴───────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (main) │ 701.36ms │
│ Total Time (upstream_coalesce) │ 676.22ms │
│ Average Time (main) │ 31.88ms │
│ Average Time (upstream_coalesce) │ 30.74ms │
│ Queries Faster │ 9 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 11 │
│ Queries with Failure │ 0 │
└──────────────────────────────────┴──────────┘
🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/test_upstream_coalesce (ed31ce17d8743c3decd2dfbcb1e4bdce36d8132b) to 0c3bb78e24722e2a4f19cdc9a76f1c498949f5be diff
Benchmarks: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
🤖: Benchmark completed
Details
Comparing HEAD and alamb_test_upstream_coalesce
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_test_upstream_coalesce ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1828.36 ms │ 1854.51 ms │ no change │
│ QQuery 1 │ 723.60 ms │ 683.35 ms │ +1.06x faster │
│ QQuery 2 │ 1384.70 ms │ 1321.27 ms │ no change │
│ QQuery 3 │ 658.78 ms │ 642.70 ms │ no change │
│ QQuery 4 │ 1342.72 ms │ 1349.52 ms │ no change │
│ QQuery 5 │ 15021.90 ms │ 15188.07 ms │ no change │
│ QQuery 6 │ 1999.39 ms │ 1980.18 ms │ no change │
│ QQuery 7 │ 1953.41 ms │ 1920.72 ms │ no change │
│ QQuery 8 │ 808.13 ms │ 797.19 ms │ no change │
└──────────────┴─────────────┴──────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 25720.98ms │
│ Total Time (alamb_test_upstream_coalesce) │ 25737.51ms │
│ Average Time (HEAD) │ 2857.89ms │
│ Average Time (alamb_test_upstream_coalesce) │ 2859.72ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 8 │
│ Queries with Failure │ 0 │
└─────────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_test_upstream_coalesce ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 15.55 ms │ 15.10 ms │ no change │
│ QQuery 1 │ 33.14 ms │ 33.18 ms │ no change │
│ QQuery 2 │ 80.86 ms │ 81.13 ms │ no change │
│ QQuery 3 │ 96.22 ms │ 98.51 ms │ no change │
│ QQuery 4 │ 593.20 ms │ 597.56 ms │ no change │
│ QQuery 5 │ 847.54 ms │ 817.57 ms │ no change │
│ QQuery 6 │ 23.11 ms │ 25.77 ms │ 1.12x slower │
│ QQuery 7 │ 35.71 ms │ 35.94 ms │ no change │
│ QQuery 8 │ 860.27 ms │ 872.05 ms │ no change │
│ QQuery 9 │ 1176.62 ms │ 1162.75 ms │ no change │
│ QQuery 10 │ 260.45 ms │ 245.56 ms │ +1.06x faster │
│ QQuery 11 │ 290.23 ms │ 273.78 ms │ +1.06x faster │
│ QQuery 12 │ 878.76 ms │ 875.99 ms │ no change │
│ QQuery 13 │ 1250.59 ms │ 1166.84 ms │ +1.07x faster │
│ QQuery 14 │ 815.43 ms │ 795.20 ms │ no change │
│ QQuery 15 │ 789.66 ms │ 806.26 ms │ no change │
│ QQuery 16 │ 1652.56 ms │ 1601.31 ms │ no change │
│ QQuery 17 │ 1631.03 ms │ 1584.09 ms │ no change │
│ QQuery 18 │ 2907.58 ms │ 2860.38 ms │ no change │
│ QQuery 19 │ 88.03 ms │ 85.04 ms │ no change │
│ QQuery 20 │ 1116.73 ms │ 1094.13 ms │ no change │
│ QQuery 21 │ 1231.96 ms │ 1243.58 ms │ no change │
│ QQuery 22 │ 2086.33 ms │ 2060.10 ms │ no change │
│ QQuery 23 │ 7609.11 ms │ 7638.07 ms │ no change │
│ QQuery 24 │ 446.74 ms │ 432.72 ms │ no change │
│ QQuery 25 │ 378.26 ms │ 364.62 ms │ no change │
│ QQuery 26 │ 517.69 ms │ 485.77 ms │ +1.07x faster │
│ QQuery 27 │ 1470.79 ms │ 1585.42 ms │ 1.08x slower │
│ QQuery 28 │ 11761.10 ms │ 12264.48 ms │ no change │
│ QQuery 29 │ 513.95 ms │ 526.86 ms │ no change │
│ QQuery 30 │ 765.22 ms │ 809.07 ms │ 1.06x slower │
│ QQuery 31 │ 825.49 ms │ 823.74 ms │ no change │
│ QQuery 32 │ 2502.42 ms │ 2499.88 ms │ no change │
│ QQuery 33 │ 3170.06 ms │ 3128.32 ms │ no change │
│ QQuery 34 │ 3162.98 ms │ 3156.29 ms │ no change │
│ QQuery 35 │ 1263.14 ms │ 1255.38 ms │ no change │
│ QQuery 36 │ 121.47 ms │ 120.76 ms │ no change │
│ QQuery 37 │ 57.52 ms │ 55.48 ms │ no change │
│ QQuery 38 │ 121.35 ms │ 119.58 ms │ no change │
│ QQuery 39 │ 188.51 ms │ 191.79 ms │ no change │
│ QQuery 40 │ 49.48 ms │ 47.64 ms │ no change │
│ QQuery 41 │ 43.56 ms │ 44.27 ms │ no change │
│ QQuery 42 │ 38.51 ms │ 38.18 ms │ no change │
└──────────────┴─────────────┴──────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 53768.90ms │
│ Total Time (alamb_test_upstream_coalesce) │ 54020.19ms │
│ Average Time (HEAD) │ 1250.44ms │
│ Average Time (alamb_test_upstream_coalesce) │ 1256.28ms │
│ Queries Faster │ 4 │
│ Queries Slower │ 3 │
│ Queries with No Change │ 36 │
│ Queries with Failure │ 0 │
└─────────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_test_upstream_coalesce ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 102.61 ms │ 92.16 ms │ +1.11x faster │
│ QQuery 2 │ 20.52 ms │ 21.48 ms │ no change │
│ QQuery 3 │ 32.05 ms │ 32.95 ms │ no change │
│ QQuery 4 │ 18.62 ms │ 19.13 ms │ no change │
│ QQuery 5 │ 49.90 ms │ 48.34 ms │ no change │
│ QQuery 6 │ 11.97 ms │ 11.81 ms │ no change │
│ QQuery 7 │ 87.76 ms │ 84.68 ms │ no change │
│ QQuery 8 │ 23.83 ms │ 24.85 ms │ no change │
│ QQuery 9 │ 54.08 ms │ 54.05 ms │ no change │
│ QQuery 10 │ 43.02 ms │ 42.77 ms │ no change │
│ QQuery 11 │ 11.23 ms │ 11.41 ms │ no change │
│ QQuery 12 │ 34.69 ms │ 33.75 ms │ no change │
│ QQuery 13 │ 26.52 ms │ 25.46 ms │ no change │
│ QQuery 14 │ 9.55 ms │ 9.97 ms │ no change │
│ QQuery 15 │ 18.85 ms │ 19.02 ms │ no change │
│ QQuery 16 │ 19.09 ms │ 17.96 ms │ +1.06x faster │
│ QQuery 17 │ 94.73 ms │ 94.57 ms │ no change │
│ QQuery 18 │ 194.78 ms │ 172.17 ms │ +1.13x faster │
│ QQuery 19 │ 25.89 ms │ 24.97 ms │ no change │
│ QQuery 20 │ 33.05 ms │ 32.02 ms │ no change │
│ QQuery 21 │ 148.87 ms │ 143.28 ms │ no change │
│ QQuery 22 │ 14.76 ms │ 14.74 ms │ no change │
└──────────────┴───────────┴──────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1076.36ms │
│ Total Time (alamb_test_upstream_coalesce) │ 1031.55ms │
│ Average Time (HEAD) │ 48.93ms │
│ Average Time (alamb_test_upstream_coalesce) │ 46.89ms │
│ Queries Faster │ 3 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 19 │
│ Queries with Failure │ 0 │
└─────────────────────────────────────────────┴───────────┘
🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/test_upstream_coalesce (c5bb25ecd54f1d026a603f615af2f3ae730003c1) to e6df27c3ba49b32fef0689c87cb787d2f6948939 diff
Benchmarks: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
🤖: Benchmark completed
Details
Comparing HEAD and alamb_test_upstream_coalesce
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_test_upstream_coalesce ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1932.11 ms │ 1959.62 ms │ no change │
│ QQuery 1 │ 671.60 ms │ 731.10 ms │ 1.09x slower │
│ QQuery 2 │ 1354.58 ms │ 1390.01 ms │ no change │
│ QQuery 3 │ 677.33 ms │ 672.01 ms │ no change │
│ QQuery 4 │ 1359.71 ms │ 1413.07 ms │ no change │
│ QQuery 5 │ 15014.69 ms │ 15070.00 ms │ no change │
│ QQuery 6 │ 2038.40 ms │ 2077.47 ms │ no change │
│ QQuery 7 │ 1899.44 ms │ 1917.93 ms │ no change │
│ QQuery 8 │ 799.21 ms │ 802.77 ms │ no change │
└──────────────┴─────────────┴──────────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 25747.08ms │
│ Total Time (alamb_test_upstream_coalesce) │ 26033.98ms │
│ Average Time (HEAD) │ 2860.79ms │
│ Average Time (alamb_test_upstream_coalesce) │ 2892.66ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 8 │
│ Queries with Failure │ 0 │
└─────────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_test_upstream_coalesce ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 15.29 ms │ 14.86 ms │ no change │
│ QQuery 1 │ 32.32 ms │ 32.47 ms │ no change │
│ QQuery 2 │ 80.82 ms │ 83.09 ms │ no change │
│ QQuery 3 │ 97.30 ms │ 100.35 ms │ no change │
│ QQuery 4 │ 582.52 ms │ 601.94 ms │ no change │
│ QQuery 5 │ 855.20 ms │ 890.36 ms │ no change │
│ QQuery 6 │ 24.39 ms │ 22.99 ms │ +1.06x faster │
│ QQuery 7 │ 37.26 ms │ 37.48 ms │ no change │
│ QQuery 8 │ 870.44 ms │ 903.78 ms │ no change │
│ QQuery 9 │ 1149.36 ms │ 1158.10 ms │ no change │
│ QQuery 10 │ 254.65 ms │ 246.37 ms │ no change │
│ QQuery 11 │ 283.28 ms │ 274.40 ms │ no change │
│ QQuery 12 │ 893.90 ms │ 869.23 ms │ no change │
│ QQuery 13 │ 1245.44 ms │ 1204.88 ms │ no change │
│ QQuery 14 │ 823.03 ms │ 807.97 ms │ no change │
│ QQuery 15 │ 784.04 ms │ 779.40 ms │ no change │
│ QQuery 16 │ 1624.44 ms │ 1598.59 ms │ no change │
│ QQuery 17 │ 1621.59 ms │ 1594.75 ms │ no change │
│ QQuery 18 │ 2897.95 ms │ 2867.76 ms │ no change │
│ QQuery 19 │ 85.75 ms │ 85.91 ms │ no change │
│ QQuery 20 │ 1137.00 ms │ 1185.54 ms │ no change │
│ QQuery 21 │ 1285.35 ms │ 1311.01 ms │ no change │
│ QQuery 22 │ 2134.45 ms │ 2186.42 ms │ no change │
│ QQuery 23 │ 7323.06 ms │ 7573.71 ms │ no change │
│ QQuery 24 │ 438.82 ms │ 427.57 ms │ no change │
│ QQuery 25 │ 309.06 ms │ 298.56 ms │ no change │
│ QQuery 26 │ 435.91 ms │ 431.97 ms │ no change │
│ QQuery 27 │ 1513.35 ms │ 1592.76 ms │ 1.05x slower │
│ QQuery 28 │ 11798.50 ms │ 12256.59 ms │ no change │
│ QQuery 29 │ 524.57 ms │ 532.43 ms │ no change │
│ QQuery 30 │ 762.01 ms │ 807.18 ms │ 1.06x slower │
│ QQuery 31 │ 776.54 ms │ 818.54 ms │ 1.05x slower │
│ QQuery 32 │ 2461.47 ms │ 2445.66 ms │ no change │
│ QQuery 33 │ 3151.15 ms │ 3163.82 ms │ no change │
│ QQuery 34 │ 3136.04 ms │ 3162.43 ms │ no change │
│ QQuery 35 │ 1282.86 ms │ 1254.00 ms │ no change │
│ QQuery 36 │ 123.18 ms │ 128.16 ms │ no change │
│ QQuery 37 │ 55.08 ms │ 57.78 ms │ no change │
│ QQuery 38 │ 120.54 ms │ 123.58 ms │ no change │
│ QQuery 39 │ 200.20 ms │ 195.98 ms │ no change │
│ QQuery 40 │ 48.71 ms │ 48.11 ms │ no change │
│ QQuery 41 │ 44.78 ms │ 43.76 ms │ no change │
│ QQuery 42 │ 38.79 ms │ 38.68 ms │ no change │
└──────────────┴─────────────┴──────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 53360.37ms │
│ Total Time (alamb_test_upstream_coalesce) │ 54258.91ms │
│ Average Time (HEAD) │ 1240.94ms │
│ Average Time (alamb_test_upstream_coalesce) │ 1261.84ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 3 │
│ Queries with No Change │ 39 │
│ Queries with Failure │ 0 │
└─────────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_test_upstream_coalesce ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 97.12 ms │ 89.91 ms │ +1.08x faster │
│ QQuery 2 │ 20.42 ms │ 21.16 ms │ no change │
│ QQuery 3 │ 32.17 ms │ 32.80 ms │ no change │
│ QQuery 4 │ 17.93 ms │ 19.10 ms │ 1.06x slower │
│ QQuery 5 │ 48.27 ms │ 49.10 ms │ no change │
│ QQuery 6 │ 11.91 ms │ 11.81 ms │ no change │
│ QQuery 7 │ 88.52 ms │ 83.99 ms │ +1.05x faster │
│ QQuery 8 │ 22.96 ms │ 23.81 ms │ no change │
│ QQuery 9 │ 53.28 ms │ 53.21 ms │ no change │
│ QQuery 10 │ 42.29 ms │ 43.34 ms │ no change │
│ QQuery 11 │ 11.12 ms │ 11.29 ms │ no change │
│ QQuery 12 │ 34.22 ms │ 33.59 ms │ no change │
│ QQuery 13 │ 26.44 ms │ 26.00 ms │ no change │
│ QQuery 14 │ 10.09 ms │ 9.87 ms │ no change │
│ QQuery 15 │ 20.10 ms │ 19.39 ms │ no change │
│ QQuery 16 │ 18.92 ms │ 18.02 ms │ no change │
│ QQuery 17 │ 96.13 ms │ 94.21 ms │ no change │
│ QQuery 18 │ 190.71 ms │ 172.82 ms │ +1.10x faster │
│ QQuery 19 │ 25.35 ms │ 26.69 ms │ 1.05x slower │
│ QQuery 20 │ 31.94 ms │ 32.32 ms │ no change │
│ QQuery 21 │ 150.45 ms │ 147.83 ms │ no change │
│ QQuery 22 │ 14.93 ms │ 14.82 ms │ no change │
└──────────────┴───────────┴──────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1065.26ms │
│ Total Time (alamb_test_upstream_coalesce) │ 1035.06ms │
│ Average Time (HEAD) │ 48.42ms │
│ Average Time (alamb_test_upstream_coalesce) │ 47.05ms │
│ Queries Faster │ 3 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 17 │
│ Queries with Failure │ 0 │
└─────────────────────────────────────────────┴───────────┘
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
@zhuqi-lucas is working on this in https://github.com/apache/datafusion/pull/17105 instead