Implement memory management in shuffle writer

[andygrove]

Is your feature request related to a problem or challenge? Please describe what you are trying to do. There is no memory management in shuffle writer

Describe the solution you'd like

Blaze implements file-based shuffle with memory management in https://github.com/blaze-init/blaze/blob/master/native-engine/datafusion-ext/src/shuffle_writer_exec.rs

Perhaps we can learn from that?

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context See discussion in https://github.com/apache/arrow-datafusion/pull/3493#issuecomment-1247587017

[mingmwang]

I will take care of this ticket.