datafusion icon indicating copy to clipboard operation
datafusion copied to clipboard

Published 20 hours ago • verified publisher icon apache

Set the default value of `datafusion.execution.collect_statistics` to `true`

Open AdamGS opened this issue 5 months ago • 5 comments

Which issue does this PR close?

  • Closes ##16158 and #16444.

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

AdamGS avatar Jun 18 '25 18:06 AdamGS

I think we should run some benchmarks here to see if the regression is gone

jonathanc-n avatar Jun 18 '25 19:06 jonathanc-n

Definitely, I'll run our benchmarks once I get all tests passing here.

AdamGS avatar Jun 18 '25 19:06 AdamGS

I'm getting a lot of sqllogictest failures, is there a reason to think there something weird going on? I was somewhat open to the idea its all fine until I ran into the last test in union_by_name.slt suddenly passing.

I'll push all of them soon so others can also take a look.

AdamGS avatar Jun 18 '25 20:06 AdamGS

Added a short upgrade note

AdamGS avatar Jun 18 '25 21:06 AdamGS

Got a similar test failure to #16448 (issue filed in #16452). I have to conclude its personal at this point, I'll try and find some time to dig into it.

AdamGS avatar Jun 18 '25 21:06 AdamGS

Our benchmarks show this change fixes the performance regression we saw - https://github.com/vortex-data/vortex/pull/3567

AdamGS avatar Jun 19 '25 00:06 AdamGS