datafusion icon indicating copy to clipboard operation
datafusion copied to clipboard
verified publisher icon apache

Improve DataFusion error handling

Open comphead opened this issue 2 years ago • 1 comments

Is your feature request related to a problem or challenge?

I'm trying to collect known issues related to error handling in datafusion.

So far I can see different categories which can be improved

  1. DataFusion heavily relies on Rust Result<T, E> and sometimes its hard to identify context where the actual error is happening, the situation becomes even worse for downstream projects.
  • [ ] https://github.com/apache/arrow-datafusion/issues/5283
  • [ ] https://github.com/apache/arrow-datafusion/issues/3410 (this related to upstream arrow-rs)
  1. Overuse panics
  • [x] https://github.com/apache/arrow-datafusion/issues/3313
  • [x] https://github.com/apache/arrow-datafusion/issues/3317
  • [ ] https://github.com/apache/arrow-datafusion/issues/3316
  1. Overuse internal error type in situations where more specific error kind can be used
  • [ ] https://github.com/apache/arrow-datafusion/issues/6108
  • [x] https://github.com/apache/arrow-datafusion/issues/9164
  1. General
  • [ ] https://github.com/apache/arrow-datafusion/issues/7552

Describe the solution you'd like

We can try to address some of the issues with implementing a generic error macros that will be used for handling DF errors. Having done that allows to process errors in standard unified way, experiment with stacktraces and other details needed to be exposed for the end user.

  • [x] https://github.com/apache/arrow-datafusion/pull/7115
  • [x] https://github.com/apache/arrow-datafusion/pull/7293
  • [x] https://github.com/apache/arrow-datafusion/pull/7340
  • [x] https://github.com/apache/arrow-datafusion/pull/7361
  • [x] https://github.com/apache/arrow-datafusion/pull/8586
  • [x] SQL Error macros
  • [ ] Parquet Error macros
  • [ ] ObjectStore Error macros
  • [ ] IOError macros
  • [ ] https://github.com/apache/arrow-datafusion/pull/8620
  • [ ] ResourceExhausted Error macros
  • [ ] External Error macros
  • [ ] Context Error macros
  • [x] Substrait Error macros
  • [x] https://github.com/apache/arrow-datafusion/pull/7434

Describe alternatives you've considered

No response

Additional context

No response

comphead avatar Aug 21 '23 23:08 comphead

I think we have made significant progress on this issue. I wonder if we still have more planned work?

alamb avatar Apr 22 '24 10:04 alamb