datafusion
datafusion copied to clipboard
apache
Use Tokio's task budget consistently, better APIs to support task cancellation
Which issue does this PR close?
- Closes #16353.
Rationale for this change
RecordBatchStreamReceiver supports cooperative scheduling implicitly by using Tokio's task budget. YieldStream currently uses a custom mechanism. It would be better to use a single mechanism consistently.
What changes are included in this PR?
- Renamed YieldStream and related types to CooperativeStream
- Removed configuration option which is no longer applicable
- Enabled cooperative scheduling in spill manager
Note that the implementation of CooperativeStream in this PR is suboptimal. The final implementation requires https://github.com/tokio-rs/tokio/pull/7405 which I'm trying to move along as best I can.
Are these changes tested?
Covered by ~infinite_cancel~coop test.
Are there any user-facing changes?
Yes, the datafusion.optimizer.yield_period configuration option is removed, but at the time of writing this has not been released yet.
Thanks for the draft -- this is inline with my understanding from your description. I think it will inch us closer to a good, lasting solution (especially after your upstream tokio also PR merges). Feel free to ping me for a more detailed review once you are done with it
@ozankabak I've pushed the optimizer rule changes I had in mind. This introduces two new execution plan properties that capture the evaluation type (how children are evaluated: eager vs lazy) and the scheduling type (how poll_next will behave wrt scheduling: blocking vs cooperative).
With those two combined the tree can be rewritten in a bottom up fashion. Every leaf that is not cooperative gets wrapped as before. Additionally, any eager evaluating nodes (i.e. exchanges) that are not cooperative are wrapped. This should ensure the entire plan participates in cooperative scheduling.
The only caveat that remains is dynamic stream creation. Operators that do that need to take the necessary precautions themselves. I already update the spill manager for this in the previous commit.
While I was writing this I started wondering if evaluation type should be a per child thing. In my spawn experiment branch for instance hash join is eager for the build side, but lazy for the probe side. Perhaps it would be best to leave room for that.
While I was writing this I started wondering if evaluation type should be a per child thing. In my spawn experiment branch for instance hash join is eager for the build side, but lazy for the probe side. Perhaps it would be best to leave room for that.
This is in alignment with what I was thinking, let's do it that way
Thinking about it some more. The evaluation type is intended to describe how the operator computes record batches itself: lazy on demand, or by driving things itself. I’m kind of trying to refer to the terminology from the volcano paper. That talks about demand-driven and data-driven operators. I had first called this 'drive type' with values 'demand' and 'data', but that felt a bit awkward. Since this is actually a property of how the operator prepares its output, one value per operator is probably fine after all.
What I'm trying to do with this is find the exchanges in the plan. The current set that's present in DataFusion is all fine, but if you were to implement one using std::sync::mpsc::channel instead of the one from tokio, explicit cooperation with the scheduler would be necessary again.
Thinking about it some more. The evaluation type is intended to describe how the operator computes record batches itself: lazy on demand, or by driving things itself. I’m kind of trying to refer to the terminology from the volcano paper. That talks about demand-driven and data-driven operators. I had first called this 'drive type' with values 'demand' and 'data', but that felt a bit awkward. Since this is actually a property of how the operator prepares its output, one value per operator is probably fine after all.
OK - let's try that way, we can change later if it turns out to be insufficient
I've added a commit to this PR that:
- Makes the tests more robust. Rather than hanging when there's an issue they will fail.
- Removes duplication from the tests
- Removes repartitions and coalesces in most tests to ensure we're actually testing the right thing. I'm to get the tests to set up the challenging situations rather than massaging things so that they work.
- Merges in some of the extra test cases I still had lying around
The current Tokio code is only an approximation of the final code. For the final version we're dependent on a Tokio PR getting merged and released. In order to unblock this PR and allow review to proceed, I've added a couple of feature flags that make the Tokio based code disabled by default. The intention is to allow comparing the two variants from a performance point-of-view and consider the revised cooperative API for merging already.
I took a quick look at this yesterday and it was looking good -- I will review in more detail in a day or two. Thanks
I've been trying to measure the impact of these changes, but for the life of me I can't get useful results out of the benchmark comparison script. The sample below is actually comparing the exact same commit on a mac mini M4 and yet I still see large fluctuations. Not sure what else I can do 🤷♂️.
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ baseline ┃ branch ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 9.71 ms │ 9.87 ms │ no change │
│ QQuery 1 │ 15.07 ms │ 15.36 ms │ no change │
│ QQuery 2 │ 47.76 ms │ 47.81 ms │ no change │
│ QQuery 3 │ 46.78 ms │ 47.13 ms │ no change │
│ QQuery 4 │ 426.60 ms │ 426.87 ms │ no change │
│ QQuery 5 │ 557.13 ms │ 559.79 ms │ no change │
│ QQuery 6 │ 11.37 ms │ 11.30 ms │ no change │
│ QQuery 7 │ 16.17 ms │ 15.99 ms │ no change │
│ QQuery 8 │ 518.13 ms │ 516.98 ms │ no change │
│ QQuery 9 │ 671.89 ms │ 669.00 ms │ no change │
│ QQuery 10 │ 146.75 ms │ 149.95 ms │ no change │
│ QQuery 11 │ 162.79 ms │ 164.02 ms │ no change │
│ QQuery 12 │ 567.28 ms │ 569.63 ms │ no change │
│ QQuery 13 │ 799.84 ms │ 796.18 ms │ no change │
│ QQuery 14 │ 542.10 ms │ 535.99 ms │ no change │
│ QQuery 15 │ 502.33 ms │ 500.89 ms │ no change │
│ QQuery 16 │ 1151.00 ms │ 1150.96 ms │ no change │
│ QQuery 17 │ 1127.41 ms │ 1116.08 ms │ no change │
│ QQuery 18 │ 2195.19 ms │ 2182.97 ms │ no change │
│ QQuery 19 │ 39.96 ms │ 40.64 ms │ no change │
│ QQuery 20 │ 858.48 ms │ 859.71 ms │ no change │
│ QQuery 21 │ 988.68 ms │ 1048.57 ms │ 1.06x slower │
│ QQuery 22 │ 1594.48 ms │ 2153.36 ms │ 1.35x slower │
│ QQuery 23 │ 6399.31 ms │ 6905.44 ms │ 1.08x slower │
│ QQuery 24 │ 305.36 ms │ 299.88 ms │ no change │
│ QQuery 25 │ 274.50 ms │ 276.00 ms │ no change │
│ QQuery 26 │ 338.09 ms │ 346.69 ms │ no change │
│ QQuery 27 │ 1140.21 ms │ 1213.95 ms │ 1.06x slower │
│ QQuery 28 │ 8881.31 ms │ 9061.62 ms │ no change │
│ QQuery 29 │ 417.02 ms │ 414.85 ms │ no change │
│ QQuery 30 │ 487.07 ms │ 486.73 ms │ no change │
│ QQuery 31 │ 507.31 ms │ 508.16 ms │ no change │
│ QQuery 32 │ 2154.40 ms │ 1917.22 ms │ +1.12x faster │
│ QQuery 33 │ 2498.71 ms │ 2467.38 ms │ no change │
│ QQuery 34 │ 2678.02 ms │ 2802.00 ms │ no change │
│ QQuery 35 │ 751.17 ms │ 772.78 ms │ no change │
│ QQuery 36 │ 61.54 ms │ 62.50 ms │ no change │
│ QQuery 37 │ 25.67 ms │ 25.56 ms │ no change │
│ QQuery 38 │ 61.80 ms │ 62.39 ms │ no change │
│ QQuery 39 │ 101.51 ms │ 102.13 ms │ no change │
│ QQuery 40 │ 18.26 ms │ 19.09 ms │ no change │
│ QQuery 41 │ 17.70 ms │ 17.77 ms │ no change │
│ QQuery 42 │ 14.84 ms │ 15.32 ms │ no change │
└──────────────┴────────────┴────────────┴───────────────┘
I've been trying to measure the impact of these changes, but for the life of me I can't get useful results out of the benchmark comparison script. The sample below is actually comparing the exact same commit on a mac mini M4 and yet I still see large fluctuations. Not sure what else I can do 🤷♂️.
I agree the existing benchmarks are not perfect -- figuring out how to make them better would be good, though as you probably know it is a challenge of benchmarking in general. I'll kick off some runs against this PR with my scripts to see what it shows.
🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing task_budget (5bfff9798b2815377ef8d40dfaffd26947c51385) to 06631c25316ed2fc20ab8114d0dcc801f353fbad diff
Benchmarks: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
I agree the existing benchmarks are not perfect
Not trying to criticize the benchmarks, just saying I'm having trouble using it as a guide.
@alamb Note that by default the current branch is using the local counter method @zhuqi-lucas implemented for YieldStream. Performance should be identical to main in other words.
🤖: Benchmark completed
Details
Comparing HEAD and task_budget
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ task_budget ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0 │ 1938.69 ms │ 1955.98 ms │ no change │
│ QQuery 1 │ 728.67 ms │ 732.71 ms │ no change │
│ QQuery 2 │ 1393.40 ms │ 1380.16 ms │ no change │
│ QQuery 3 │ 654.25 ms │ 667.02 ms │ no change │
│ QQuery 4 │ 1374.93 ms │ 1368.90 ms │ no change │
│ QQuery 5 │ 14843.62 ms │ 14967.16 ms │ no change │
│ QQuery 6 │ 2001.52 ms │ 2082.06 ms │ no change │
│ QQuery 7 │ 1889.83 ms │ 1950.81 ms │ no change │
│ QQuery 8 │ 805.12 ms │ 809.25 ms │ no change │
└──────────────┴─────────────┴─────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 25630.03ms │
│ Total Time (task_budget) │ 25914.06ms │
│ Average Time (HEAD) │ 2847.78ms │
│ Average Time (task_budget) │ 2879.34ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 9 │
│ Queries with Failure │ 0 │
└────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ task_budget ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 15.26 ms │ 15.20 ms │ no change │
│ QQuery 1 │ 33.51 ms │ 32.70 ms │ no change │
│ QQuery 2 │ 81.78 ms │ 80.44 ms │ no change │
│ QQuery 3 │ 100.21 ms │ 96.71 ms │ no change │
│ QQuery 4 │ 578.10 ms │ 585.73 ms │ no change │
│ QQuery 5 │ 838.83 ms │ 887.89 ms │ 1.06x slower │
│ QQuery 6 │ 24.33 ms │ 22.32 ms │ +1.09x faster │
│ QQuery 7 │ 38.56 ms │ 36.90 ms │ no change │
│ QQuery 8 │ 850.10 ms │ 856.20 ms │ no change │
│ QQuery 9 │ 1151.01 ms │ 1146.42 ms │ no change │
│ QQuery 10 │ 252.88 ms │ 263.74 ms │ no change │
│ QQuery 11 │ 283.58 ms │ 290.15 ms │ no change │
│ QQuery 12 │ 869.70 ms │ 866.73 ms │ no change │
│ QQuery 13 │ 1255.69 ms │ 1286.10 ms │ no change │
│ QQuery 14 │ 809.83 ms │ 812.02 ms │ no change │
│ QQuery 15 │ 777.07 ms │ 786.77 ms │ no change │
│ QQuery 16 │ 1613.49 ms │ 1629.44 ms │ no change │
│ QQuery 17 │ 1567.97 ms │ 1580.52 ms │ no change │
│ QQuery 18 │ 2886.23 ms │ 2891.01 ms │ no change │
│ QQuery 19 │ 87.29 ms │ 84.38 ms │ no change │
│ QQuery 20 │ 1113.26 ms │ 1152.62 ms │ no change │
│ QQuery 21 │ 1252.92 ms │ 1328.12 ms │ 1.06x slower │
│ QQuery 22 │ 2069.04 ms │ 2168.76 ms │ no change │
│ QQuery 23 │ 7539.31 ms │ 7957.48 ms │ 1.06x slower │
│ QQuery 24 │ 443.12 ms │ 464.35 ms │ no change │
│ QQuery 25 │ 379.12 ms │ 404.81 ms │ 1.07x slower │
│ QQuery 26 │ 503.89 ms │ 530.15 ms │ 1.05x slower │
│ QQuery 27 │ 1500.52 ms │ 1566.86 ms │ no change │
│ QQuery 28 │ 12013.51 ms │ 11817.68 ms │ no change │
│ QQuery 29 │ 527.40 ms │ 525.63 ms │ no change │
│ QQuery 30 │ 758.72 ms │ 791.22 ms │ no change │
│ QQuery 31 │ 811.10 ms │ 829.72 ms │ no change │
│ QQuery 32 │ 2485.30 ms │ 2553.71 ms │ no change │
│ QQuery 33 │ 3181.91 ms │ 3185.99 ms │ no change │
│ QQuery 34 │ 3176.40 ms │ 3209.67 ms │ no change │
│ QQuery 35 │ 1228.64 ms │ 1223.48 ms │ no change │
│ QQuery 36 │ 120.17 ms │ 127.76 ms │ 1.06x slower │
│ QQuery 37 │ 57.02 ms │ 56.75 ms │ no change │
│ QQuery 38 │ 120.93 ms │ 127.28 ms │ 1.05x slower │
│ QQuery 39 │ 191.04 ms │ 202.25 ms │ 1.06x slower │
│ QQuery 40 │ 47.95 ms │ 47.32 ms │ no change │
│ QQuery 41 │ 45.18 ms │ 45.53 ms │ no change │
│ QQuery 42 │ 36.63 ms │ 39.44 ms │ 1.08x slower │
└──────────────┴─────────────┴─────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 53718.51ms │
│ Total Time (task_budget) │ 54607.96ms │
│ Average Time (HEAD) │ 1249.27ms │
│ Average Time (task_budget) │ 1269.95ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 9 │
│ Queries with No Change │ 33 │
│ Queries with Failure │ 0 │
└────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ task_budget ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 101.57 ms │ 101.26 ms │ no change │
│ QQuery 2 │ 21.12 ms │ 21.41 ms │ no change │
│ QQuery 3 │ 32.07 ms │ 32.64 ms │ no change │
│ QQuery 4 │ 18.53 ms │ 18.48 ms │ no change │
│ QQuery 5 │ 48.96 ms │ 49.09 ms │ no change │
│ QQuery 6 │ 11.91 ms │ 11.82 ms │ no change │
│ QQuery 7 │ 87.54 ms │ 89.53 ms │ no change │
│ QQuery 8 │ 25.52 ms │ 24.90 ms │ no change │
│ QQuery 9 │ 54.10 ms │ 55.00 ms │ no change │
│ QQuery 10 │ 42.23 ms │ 43.43 ms │ no change │
│ QQuery 11 │ 11.61 ms │ 11.44 ms │ no change │
│ QQuery 12 │ 34.46 ms │ 34.10 ms │ no change │
│ QQuery 13 │ 26.49 ms │ 26.42 ms │ no change │
│ QQuery 14 │ 9.60 ms │ 9.81 ms │ no change │
│ QQuery 15 │ 19.11 ms │ 18.77 ms │ no change │
│ QQuery 16 │ 18.76 ms │ 18.83 ms │ no change │
│ QQuery 17 │ 95.76 ms │ 94.69 ms │ no change │
│ QQuery 18 │ 199.88 ms │ 187.28 ms │ +1.07x faster │
│ QQuery 19 │ 25.75 ms │ 25.18 ms │ no change │
│ QQuery 20 │ 31.96 ms │ 31.34 ms │ no change │
│ QQuery 21 │ 148.19 ms │ 146.33 ms │ no change │
│ QQuery 22 │ 15.24 ms │ 14.98 ms │ no change │
└──────────────┴───────────┴─────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1080.35ms │
│ Total Time (task_budget) │ 1066.72ms │
│ Average Time (HEAD) │ 49.11ms │
│ Average Time (task_budget) │ 48.49ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 21 │
│ Queries with Failure │ 0 │
└────────────────────────────┴───────────┘
🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing task_budget (a749e8f60cbcdf5542b4b69a0a2f94c7790fab45) to 06631c25316ed2fc20ab8114d0dcc801f353fbad diff
Benchmarks: clickbench_1
Results will be posted here when complete
🤖: Benchmark completed
Details
Comparing HEAD and task_budget
--------------------
Benchmark clickbench_1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ task_budget ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 51.47 ms │ 46.15 ms │ +1.12x faster │
│ QQuery 1 │ 75.49 ms │ 72.84 ms │ no change │
│ QQuery 2 │ 109.24 ms │ 108.00 ms │ no change │
│ QQuery 3 │ 127.42 ms │ 119.32 ms │ +1.07x faster │
│ QQuery 4 │ 633.00 ms │ 685.11 ms │ 1.08x slower │
│ QQuery 5 │ 853.72 ms │ 863.05 ms │ no change │
│ QQuery 6 │ 56.74 ms │ 54.54 ms │ no change │
│ QQuery 7 │ 80.42 ms │ 82.07 ms │ no change │
│ QQuery 8 │ 881.04 ms │ 884.51 ms │ no change │
│ QQuery 9 │ 1194.27 ms │ 1202.55 ms │ no change │
│ QQuery 10 │ 300.62 ms │ 297.89 ms │ no change │
│ QQuery 11 │ 318.31 ms │ 323.44 ms │ no change │
│ QQuery 12 │ 879.94 ms │ 888.82 ms │ no change │
│ QQuery 13 │ 1245.61 ms │ 1252.15 ms │ no change │
│ QQuery 14 │ 803.27 ms │ 814.41 ms │ no change │
│ QQuery 15 │ 803.96 ms │ 822.93 ms │ no change │
│ QQuery 16 │ 1630.13 ms │ 1686.71 ms │ no change │
│ QQuery 17 │ 1584.79 ms │ 1608.29 ms │ no change │
│ QQuery 18 │ 2894.48 ms │ 2899.79 ms │ no change │
│ QQuery 19 │ 125.96 ms │ 119.63 ms │ +1.05x faster │
│ QQuery 20 │ 1140.75 ms │ 1206.46 ms │ 1.06x slower │
│ QQuery 21 │ 1337.94 ms │ 1390.68 ms │ no change │
│ QQuery 22 │ 2323.74 ms │ 2408.18 ms │ no change │
│ QQuery 23 │ 7822.32 ms │ 8103.67 ms │ no change │
│ QQuery 24 │ 489.54 ms │ 507.76 ms │ no change │
│ QQuery 25 │ 406.25 ms │ 432.43 ms │ 1.06x slower │
│ QQuery 26 │ 545.86 ms │ 568.63 ms │ no change │
│ QQuery 27 │ 1638.18 ms │ 1683.81 ms │ no change │
│ QQuery 28 │ 12379.17 ms │ 12511.87 ms │ no change │
│ QQuery 29 │ 565.53 ms │ 581.52 ms │ no change │
│ QQuery 30 │ 803.29 ms │ 816.97 ms │ no change │
│ QQuery 31 │ 843.85 ms │ 872.76 ms │ no change │
│ QQuery 32 │ 2547.32 ms │ 2563.59 ms │ no change │
│ QQuery 33 │ 3256.21 ms │ 3307.21 ms │ no change │
│ QQuery 34 │ 3266.60 ms │ 3323.18 ms │ no change │
│ QQuery 35 │ 1258.28 ms │ 1288.71 ms │ no change │
│ QQuery 36 │ 173.64 ms │ 181.16 ms │ no change │
│ QQuery 37 │ 101.14 ms │ 101.04 ms │ no change │
│ QQuery 38 │ 171.00 ms │ 179.89 ms │ 1.05x slower │
│ QQuery 39 │ 252.45 ms │ 266.95 ms │ 1.06x slower │
│ QQuery 40 │ 70.71 ms │ 88.09 ms │ 1.25x slower │
│ QQuery 41 │ 87.06 ms │ 86.11 ms │ no change │
│ QQuery 42 │ 78.55 ms │ 78.27 ms │ no change │
└──────────────┴─────────────┴─────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 56209.26ms │
│ Total Time (task_budget) │ 57381.15ms │
│ Average Time (HEAD) │ 1307.19ms │
│ Average Time (task_budget) │ 1334.45ms │
│ Queries Faster │ 3 │
│ Queries Slower │ 6 │
│ Queries with No Change │ 34 │
│ Queries with Failure │ 0 │
└────────────────────────────┴────────────┘
🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing task_budget (a749e8f60cbcdf5542b4b69a0a2f94c7790fab45) to 06631c25316ed2fc20ab8114d0dcc801f353fbad diff
Benchmarks: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
🤖: Benchmark completed
Details
Comparing HEAD and task_budget
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ task_budget ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1929.02 ms │ 1971.93 ms │ no change │
│ QQuery 1 │ 704.63 ms │ 718.11 ms │ no change │
│ QQuery 2 │ 1412.62 ms │ 1368.78 ms │ no change │
│ QQuery 3 │ 647.44 ms │ 680.23 ms │ 1.05x slower │
│ QQuery 4 │ 1400.23 ms │ 1358.38 ms │ no change │
│ QQuery 5 │ 14995.43 ms │ 15226.83 ms │ no change │
│ QQuery 6 │ 1985.14 ms │ 2087.35 ms │ 1.05x slower │
│ QQuery 7 │ 1989.35 ms │ 1969.02 ms │ no change │
│ QQuery 8 │ 785.58 ms │ 786.62 ms │ no change │
└──────────────┴─────────────┴─────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 25849.44ms │
│ Total Time (task_budget) │ 26167.24ms │
│ Average Time (HEAD) │ 2872.16ms │
│ Average Time (task_budget) │ 2907.47ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 7 │
│ Queries with Failure │ 0 │
└────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ task_budget ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 15.70 ms │ 15.32 ms │ no change │
│ QQuery 1 │ 33.40 ms │ 32.29 ms │ no change │
│ QQuery 2 │ 78.71 ms │ 81.60 ms │ no change │
│ QQuery 3 │ 94.83 ms │ 96.30 ms │ no change │
│ QQuery 4 │ 590.16 ms │ 586.19 ms │ no change │
│ QQuery 5 │ 839.78 ms │ 884.55 ms │ 1.05x slower │
│ QQuery 6 │ 23.62 ms │ 24.22 ms │ no change │
│ QQuery 7 │ 35.97 ms │ 35.71 ms │ no change │
│ QQuery 8 │ 845.86 ms │ 836.34 ms │ no change │
│ QQuery 9 │ 1155.51 ms │ 1124.37 ms │ no change │
│ QQuery 10 │ 255.94 ms │ 258.01 ms │ no change │
│ QQuery 11 │ 283.03 ms │ 285.17 ms │ no change │
│ QQuery 12 │ 861.95 ms │ 884.44 ms │ no change │
│ QQuery 13 │ 1256.74 ms │ 1274.54 ms │ no change │
│ QQuery 14 │ 801.84 ms │ 819.99 ms │ no change │
│ QQuery 15 │ 786.28 ms │ 775.94 ms │ no change │
│ QQuery 16 │ 1616.39 ms │ 1622.78 ms │ no change │
│ QQuery 17 │ 1591.05 ms │ 1593.83 ms │ no change │
│ QQuery 18 │ 2857.24 ms │ 2902.54 ms │ no change │
│ QQuery 19 │ 89.43 ms │ 84.53 ms │ +1.06x faster │
│ QQuery 20 │ 1125.85 ms │ 1171.56 ms │ no change │
│ QQuery 21 │ 1266.95 ms │ 1307.77 ms │ no change │
│ QQuery 22 │ 2089.10 ms │ 2189.53 ms │ no change │
│ QQuery 23 │ 7574.01 ms │ 7914.18 ms │ no change │
│ QQuery 24 │ 444.06 ms │ 466.35 ms │ 1.05x slower │
│ QQuery 25 │ 378.44 ms │ 397.31 ms │ no change │
│ QQuery 26 │ 521.01 ms │ 525.65 ms │ no change │
│ QQuery 27 │ 1503.49 ms │ 1548.92 ms │ no change │
│ QQuery 28 │ 11900.61 ms │ 11913.35 ms │ no change │
│ QQuery 29 │ 534.80 ms │ 540.04 ms │ no change │
│ QQuery 30 │ 764.78 ms │ 790.34 ms │ no change │
│ QQuery 31 │ 812.47 ms │ 825.30 ms │ no change │
│ QQuery 32 │ 2541.75 ms │ 2545.00 ms │ no change │
│ QQuery 33 │ 3179.10 ms │ 3206.02 ms │ no change │
│ QQuery 34 │ 3202.13 ms │ 3242.56 ms │ no change │
│ QQuery 35 │ 1219.46 ms │ 1221.75 ms │ no change │
│ QQuery 36 │ 121.25 ms │ 126.01 ms │ no change │
│ QQuery 37 │ 54.86 ms │ 56.17 ms │ no change │
│ QQuery 38 │ 119.17 ms │ 125.52 ms │ 1.05x slower │
│ QQuery 39 │ 192.11 ms │ 198.37 ms │ no change │
│ QQuery 40 │ 47.73 ms │ 50.41 ms │ 1.06x slower │
│ QQuery 41 │ 43.85 ms │ 45.09 ms │ no change │
│ QQuery 42 │ 37.48 ms │ 38.16 ms │ no change │
└──────────────┴─────────────┴─────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 53787.93ms │
│ Total Time (task_budget) │ 54664.02ms │
│ Average Time (HEAD) │ 1250.88ms │
│ Average Time (task_budget) │ 1271.26ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 4 │
│ Queries with No Change │ 38 │
│ Queries with Failure │ 0 │
└────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ task_budget ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 102.04 ms │ 101.87 ms │ no change │
│ QQuery 2 │ 21.73 ms │ 20.67 ms │ no change │
│ QQuery 3 │ 32.83 ms │ 32.19 ms │ no change │
│ QQuery 4 │ 18.89 ms │ 18.39 ms │ no change │
│ QQuery 5 │ 50.60 ms │ 49.54 ms │ no change │
│ QQuery 6 │ 12.02 ms │ 11.88 ms │ no change │
│ QQuery 7 │ 90.16 ms │ 89.23 ms │ no change │
│ QQuery 8 │ 26.40 ms │ 25.02 ms │ +1.06x faster │
│ QQuery 9 │ 54.48 ms │ 54.78 ms │ no change │
│ QQuery 10 │ 44.56 ms │ 42.64 ms │ no change │
│ QQuery 11 │ 11.60 ms │ 11.27 ms │ no change │
│ QQuery 12 │ 35.11 ms │ 34.35 ms │ no change │
│ QQuery 13 │ 26.89 ms │ 26.54 ms │ no change │
│ QQuery 14 │ 10.12 ms │ 9.75 ms │ no change │
│ QQuery 15 │ 19.80 ms │ 19.82 ms │ no change │
│ QQuery 16 │ 18.99 ms │ 18.85 ms │ no change │
│ QQuery 17 │ 97.96 ms │ 94.41 ms │ no change │
│ QQuery 18 │ 214.13 ms │ 197.52 ms │ +1.08x faster │
│ QQuery 19 │ 26.66 ms │ 25.60 ms │ no change │
│ QQuery 20 │ 32.51 ms │ 32.08 ms │ no change │
│ QQuery 21 │ 146.48 ms │ 146.42 ms │ no change │
│ QQuery 22 │ 15.46 ms │ 15.27 ms │ no change │
└──────────────┴───────────┴─────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1109.41ms │
│ Total Time (task_budget) │ 1078.08ms │
│ Average Time (HEAD) │ 50.43ms │
│ Average Time (task_budget) │ 49.00ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 20 │
│ Queries with Failure │ 0 │
└────────────────────────────┴───────────┘
I agree the existing benchmarks are not perfect
Not trying to criticize the benchmarks, just saying I'm having trouble using it as a guide. @alamb Note that by default the current branch is using the local counter method @zhuqi-lucas implemented for
YieldStream. Performance should be identical tomainin other words.
I totally get it -- we'll keep improving the benchmarks / hopefully making them easier to use / more stable over time
@alamb the remaining open question I have related to this PR is what the default implementation should be for the time being. Do we use the tokio has_budget_remaining/consume_budget approximation which is more robust or do we stick with the per-operator counter we have on main but which doesn't fix the interleave test case (marked as ignore to not fail the build)?
The current state of the PR uses the latter unless you compile with --cfg datafusion_coop=tokio_fallback.
@alamb the remaining open question I have related to this PR is what the default implementation should be for the time being. Do we use the tokio has_budget_remaining/consume_budget approximation which is more robust or do we stick with the per-operator counter we have on main but which doesn't fix the interleave test case (marked as ignore to not fail the build)?
What do you suggest?
What do you suggest?
@ozankabak I'm not sure. I've been trying to convince myself that the task budget variant is good enough for now and doesn't incur a performance penalty, but I haven't been having much luck gathering proof. It's hard to make an informed decision if you can't measure.
I think we can go ahead with the task budget -- we can revert to the alternative if we discover a (surprising) performance issue
I think we can go ahead with the task budget -- we can revert to the alternative if we discover a (surprising) performance issue
Yeah, I agree
In general I think defaulting to the current behavior (which was no extra consideration for cooperation) is also a safe thing and we can annotate additional operators we discover have issues
Just saw the merge conflict. I'll rebase.
@alamb I've changed the default to the tokio-based temporary placeholder. The version that's not using poll_proceed yet. Would you mind scheduling one last benchmark run?
🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing task_budget (3b33f79dda896390aa1d174befcfd4f8cc34033f) to 1429c92474238a91a09f1cd4a68c19d03329b6a7 diff
Benchmarks: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
🤖: Benchmark completed
Details
Comparing HEAD and task_budget
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ task_budget ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1949.96 ms │ 1930.13 ms │ no change │
│ QQuery 1 │ 675.49 ms │ 729.71 ms │ 1.08x slower │
│ QQuery 2 │ 1340.52 ms │ 1378.26 ms │ no change │
│ QQuery 3 │ 662.34 ms │ 658.36 ms │ no change │
│ QQuery 4 │ 1373.39 ms │ 1343.97 ms │ no change │
│ QQuery 5 │ 15044.29 ms │ 14931.43 ms │ no change │
│ QQuery 6 │ 2031.52 ms │ 2100.06 ms │ no change │
│ QQuery 7 │ 1949.16 ms │ 1911.84 ms │ no change │
│ QQuery 8 │ 790.17 ms │ 811.49 ms │ no change │
└──────────────┴─────────────┴─────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 25816.84ms │
│ Total Time (task_budget) │ 25795.26ms │
│ Average Time (HEAD) │ 2868.54ms │
│ Average Time (task_budget) │ 2866.14ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 8 │
│ Queries with Failure │ 0 │
└────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ task_budget ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0 │ 15.07 ms │ 15.52 ms │ no change │
│ QQuery 1 │ 32.39 ms │ 33.64 ms │ no change │
│ QQuery 2 │ 81.66 ms │ 81.29 ms │ no change │
│ QQuery 3 │ 97.11 ms │ 97.56 ms │ no change │
│ QQuery 4 │ 591.09 ms │ 593.49 ms │ no change │
│ QQuery 5 │ 821.63 ms │ 872.05 ms │ 1.06x slower │
│ QQuery 6 │ 22.34 ms │ 22.99 ms │ no change │
│ QQuery 7 │ 36.75 ms │ 36.65 ms │ no change │
│ QQuery 8 │ 844.36 ms │ 865.78 ms │ no change │
│ QQuery 9 │ 1143.96 ms │ 1148.17 ms │ no change │
│ QQuery 10 │ 253.88 ms │ 253.51 ms │ no change │
│ QQuery 11 │ 277.72 ms │ 288.38 ms │ no change │
│ QQuery 12 │ 851.39 ms │ 871.27 ms │ no change │
│ QQuery 13 │ 1121.16 ms │ 1237.87 ms │ 1.10x slower │
│ QQuery 14 │ 807.20 ms │ 810.33 ms │ no change │
│ QQuery 15 │ 740.12 ms │ 757.64 ms │ no change │
│ QQuery 16 │ 1601.02 ms │ 1610.53 ms │ no change │
│ QQuery 17 │ 1610.29 ms │ 1607.44 ms │ no change │
│ QQuery 18 │ 2862.22 ms │ 2887.82 ms │ no change │
│ QQuery 19 │ 81.40 ms │ 85.86 ms │ 1.05x slower │
│ QQuery 20 │ 1137.87 ms │ 1173.96 ms │ no change │
│ QQuery 21 │ 1276.87 ms │ 1312.58 ms │ no change │
│ QQuery 22 │ 2134.80 ms │ 2210.32 ms │ no change │
│ QQuery 23 │ 7399.34 ms │ 7585.73 ms │ no change │
│ QQuery 24 │ 434.38 ms │ 454.37 ms │ no change │
│ QQuery 25 │ 304.06 ms │ 312.91 ms │ no change │
│ QQuery 26 │ 434.86 ms │ 458.44 ms │ 1.05x slower │
│ QQuery 27 │ 1525.86 ms │ 1576.08 ms │ no change │
│ QQuery 28 │ 11727.04 ms │ 12045.71 ms │ no change │
│ QQuery 29 │ 524.27 ms │ 516.39 ms │ no change │
│ QQuery 30 │ 770.12 ms │ 793.40 ms │ no change │
│ QQuery 31 │ 801.85 ms │ 824.47 ms │ no change │
│ QQuery 32 │ 2409.23 ms │ 2419.15 ms │ no change │
│ QQuery 33 │ 3090.45 ms │ 3143.57 ms │ no change │
│ QQuery 34 │ 3152.12 ms │ 3158.17 ms │ no change │
│ QQuery 35 │ 1183.10 ms │ 1238.98 ms │ no change │
│ QQuery 36 │ 129.34 ms │ 125.39 ms │ no change │
│ QQuery 37 │ 54.74 ms │ 57.12 ms │ no change │
│ QQuery 38 │ 122.62 ms │ 125.25 ms │ no change │
│ QQuery 39 │ 199.54 ms │ 195.83 ms │ no change │
│ QQuery 40 │ 47.79 ms │ 49.30 ms │ no change │
│ QQuery 41 │ 44.60 ms │ 44.50 ms │ no change │
│ QQuery 42 │ 39.35 ms │ 38.99 ms │ no change │
└──────────────┴─────────────┴─────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 52836.93ms │
│ Total Time (task_budget) │ 54038.40ms │
│ Average Time (HEAD) │ 1228.77ms │
│ Average Time (task_budget) │ 1256.71ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 4 │
│ Queries with No Change │ 39 │
│ Queries with Failure │ 0 │
└────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ task_budget ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1 │ 102.55 ms │ 99.10 ms │ no change │
│ QQuery 2 │ 20.94 ms │ 20.73 ms │ no change │
│ QQuery 3 │ 32.32 ms │ 32.03 ms │ no change │
│ QQuery 4 │ 18.53 ms │ 18.75 ms │ no change │
│ QQuery 5 │ 48.91 ms │ 48.40 ms │ no change │
│ QQuery 6 │ 11.91 ms │ 11.84 ms │ no change │
│ QQuery 7 │ 87.18 ms │ 87.96 ms │ no change │
│ QQuery 8 │ 25.12 ms │ 24.58 ms │ no change │
│ QQuery 9 │ 53.78 ms │ 55.14 ms │ no change │
│ QQuery 10 │ 43.11 ms │ 42.77 ms │ no change │
│ QQuery 11 │ 11.14 ms │ 11.34 ms │ no change │
│ QQuery 12 │ 34.92 ms │ 34.46 ms │ no change │
│ QQuery 13 │ 25.68 ms │ 25.61 ms │ no change │
│ QQuery 14 │ 9.52 ms │ 9.68 ms │ no change │
│ QQuery 15 │ 19.06 ms │ 18.70 ms │ no change │
│ QQuery 16 │ 18.70 ms │ 18.64 ms │ no change │
│ QQuery 17 │ 93.82 ms │ 96.06 ms │ no change │
│ QQuery 18 │ 194.66 ms │ 194.28 ms │ no change │
│ QQuery 19 │ 25.27 ms │ 25.00 ms │ no change │
│ QQuery 20 │ 32.21 ms │ 31.37 ms │ no change │
│ QQuery 21 │ 144.09 ms │ 146.38 ms │ no change │
│ QQuery 22 │ 15.76 ms │ 15.12 ms │ no change │
└──────────────┴───────────┴─────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1069.18ms │
│ Total Time (task_budget) │ 1067.95ms │
│ Average Time (HEAD) │ 48.60ms │
│ Average Time (task_budget) │ 48.54ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 22 │
│ Queries with Failure │ 0 │
└────────────────────────────┴───────────┘
🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing task_budget (3b33f79dda896390aa1d174befcfd4f8cc34033f) to 1429c92474238a91a09f1cd4a68c19d03329b6a7 diff
Benchmarks: clickbench_1
Results will be posted here when complete
🤖: Benchmark completed
Details
Comparing HEAD and task_budget
--------------------
Benchmark clickbench_1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ task_budget ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 45.55 ms │ 47.02 ms │ no change │
│ QQuery 1 │ 70.90 ms │ 74.64 ms │ 1.05x slower │
│ QQuery 2 │ 108.60 ms │ 108.42 ms │ no change │
│ QQuery 3 │ 118.38 ms │ 117.48 ms │ no change │
│ QQuery 4 │ 614.68 ms │ 702.11 ms │ 1.14x slower │
│ QQuery 5 │ 843.56 ms │ 868.27 ms │ no change │
│ QQuery 6 │ 53.99 ms │ 54.51 ms │ no change │
│ QQuery 7 │ 78.46 ms │ 78.61 ms │ no change │
│ QQuery 8 │ 887.11 ms │ 904.12 ms │ no change │
│ QQuery 9 │ 1173.21 ms │ 1225.36 ms │ no change │
│ QQuery 10 │ 288.90 ms │ 287.56 ms │ no change │
│ QQuery 11 │ 313.83 ms │ 324.76 ms │ no change │
│ QQuery 12 │ 855.16 ms │ 892.54 ms │ no change │
│ QQuery 13 │ 1219.67 ms │ 1250.80 ms │ no change │
│ QQuery 14 │ 814.84 ms │ 816.85 ms │ no change │
│ QQuery 15 │ 795.60 ms │ 809.07 ms │ no change │
│ QQuery 16 │ 1603.15 ms │ 1642.79 ms │ no change │
│ QQuery 17 │ 1611.43 ms │ 1620.07 ms │ no change │
│ QQuery 18 │ 2884.61 ms │ 2906.00 ms │ no change │
│ QQuery 19 │ 121.20 ms │ 121.38 ms │ no change │
│ QQuery 20 │ 1180.69 ms │ 1189.69 ms │ no change │
│ QQuery 21 │ 1371.20 ms │ 1394.64 ms │ no change │
│ QQuery 22 │ 2379.76 ms │ 2412.27 ms │ no change │
│ QQuery 23 │ 7945.62 ms │ 8090.22 ms │ no change │
│ QQuery 24 │ 460.30 ms │ 470.69 ms │ no change │
│ QQuery 25 │ 346.40 ms │ 347.84 ms │ no change │
│ QQuery 26 │ 451.87 ms │ 477.27 ms │ 1.06x slower │
│ QQuery 27 │ 1650.72 ms │ 1684.69 ms │ no change │
│ QQuery 28 │ 12400.61 ms │ 12486.22 ms │ no change │
│ QQuery 29 │ 568.75 ms │ 554.71 ms │ no change │
│ QQuery 30 │ 797.26 ms │ 806.62 ms │ no change │
│ QQuery 31 │ 822.99 ms │ 854.98 ms │ no change │
│ QQuery 32 │ 2427.54 ms │ 2465.81 ms │ no change │
│ QQuery 33 │ 3173.31 ms │ 3268.38 ms │ no change │
│ QQuery 34 │ 3262.48 ms │ 3273.74 ms │ no change │
│ QQuery 35 │ 1259.76 ms │ 1276.50 ms │ no change │
│ QQuery 36 │ 167.14 ms │ 173.11 ms │ no change │
│ QQuery 37 │ 98.87 ms │ 106.56 ms │ 1.08x slower │
│ QQuery 38 │ 173.53 ms │ 171.48 ms │ no change │
│ QQuery 39 │ 250.93 ms │ 254.95 ms │ no change │
│ QQuery 40 │ 73.28 ms │ 80.35 ms │ 1.10x slower │
│ QQuery 41 │ 83.66 ms │ 79.01 ms │ +1.06x faster │
│ QQuery 42 │ 76.39 ms │ 74.95 ms │ no change │
└──────────────┴─────────────┴─────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 55925.88ms │
│ Total Time (task_budget) │ 56847.01ms │
│ Average Time (HEAD) │ 1300.60ms │
│ Average Time (task_budget) │ 1322.02ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 5 │
│ Queries with No Change │ 37 │
│ Queries with Failure │ 0 │
└────────────────────────────┴────────────┘