Daniël Heres comments

Results 167 comments of


                                            Daniël Heres

Upgrade DataFusion from 31.0.0 to 40.1.0

I think @Tmonster runs them once in a while

Upgrade DataFusion from 31.0.0 to 40.1.0

Thanks @Tmonster we probably make a PR soon for DataFusion 42, as many performance improvements were added compared to 41 and DataFusion will perform better for aggregations.

Create empty shuffle read task

Is the issue that we can remove tasks/partitions/stages that are empty or is it a bug?

Remove `Alias` from `Expr`

FYI @alamb @houqp

Remove `Alias` from `Expr`

Hi @jackwener . The idea is that any expression within DataFusion receives a name, so the nodes in a LogicalPlan use the `NamedExpr` type. The name in this type is...

feat: use spawned tasks to reduce call stack depth and avoid busy waiting

In what situations would these changes lead to better performance? I.e. why is query 28 28: ~ 1.10x faster?

feat: use spawned tasks to reduce call stack depth and avoid busy waiting

(Or is it just benchmark noise?)

feat: implement GroupsAccumulator for `count(DISTINCT)` aggr

It would be worthwhile to run the `clickbench_extended` benchmarks as well (`./bench.sh run clickbench_extended`)

Support broadcast exchange

@mingmwang I think for broadcasting exchange the same thing applies as normal exchanges, they are spilled to disk by default and might be maintained in memory if memory budget allows....

Support broadcast exchange

That's a good observation @mingmwang ! The difference with CollectLeft is that that mode collects the left side to one partition, whereas with broadcast we would broadcast the output of...