datafusion Add compression option to SpillManager

Which issue does this PR close?

Closes #16130 .

Further Jobs to to do

I filed a separate issue to investigate the benefit of this compression option. https://github.com/apache/datafusion/issues/16367

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

yes, new datafusion.execution config option

Jun 06 '25 05:06 ding-young

This is ready for review :) @2010YOUY01

Jun 11 '25 05:06 ding-young

Currently CI fails, but I think that is due to change introduced in another pr.

Jun 18 '25 01:06 ding-young

close and reopen to trigger CI again

Jun 18 '25 08:06 2010YOUY01

It's better to mention the option in DF49 upgrade doc

Thank you for the reminder @xudong963 .

I got a question: as I understand it, upgrade guide is like 'here is the API changes that might break your system during upgrades', and this PR is like a new feature you might want to try in the new release -- do we have a separate place to document new features like this? 🤔

Jun 19 '25 12:06 2010YOUY01

'here is the API changes that might break your system during upgrades', and this PR is like a new feature you might want to try in the new release -- do we have a separate place to document new features like this? 🤔

I was thinking the upgrade doc contains all things that we wanna highlight and let users be aware of

Jun 20 '25 07:06 xudong963

'here is the API changes that might break your system during upgrades', and this PR is like a new feature you might want to try in the new release -- do we have a separate place to document new features like this? 🤔

I was thinking the upgrade doc contains all things that we wanna highlight and let users be aware of

I see. Maybe we can put new features and API changes into different sub sections.

Jun 20 '25 08:06 2010YOUY01