datafusion-ballista icon indicating copy to clipboard operation
datafusion-ballista copied to clipboard

Consider using `with_skip_validation` for shuffle file reading

Open milenkovicm opened this issue 10 months ago • 2 comments

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Improve shuffle reading, with https://github.com/apache/arrow-rs/pull/7120 merged consider using it for shuffle files.

Describe the solution you'd like

Change shuffle reader to utilise new api.

Describe alternatives you've considered

Keep everything as it is

Additional context

  • we could try to address #944 at the same time

milenkovicm avatar Feb 27 '25 12:02 milenkovicm

/take

westhide avatar Mar 28 '25 12:03 westhide