spark icon indicating copy to clipboard operation
spark copied to clipboard

[SPARK-40488] Do not wrap exceptions thrown when datasource write fails

Open bozhang2820 opened this issue 3 years ago • 1 comments

What changes were proposed in this pull request?

Exceptions thrown when writing to datasources (e.g. FileFormatWriter.write and WriteToDataSourceV2.writeWithV2), are wrapped with SparkException("Job aborted.") or SparkException("Writing job aborted").

This provides little extra information, but generates a long stacktrace and make the debugging process more difficult.

This change removes the wrapping, and the unused error class WRITING_JOB_ABORTED.

Why are the changes needed?

This is to simplify the stacktrace thrown when writing to datasource fails.

Does this PR introduce any user-facing change?

When exceptions are thrown in datasources writes, the wrapping with SparkExceptions will be removed and stacktraces will be simplified.

How was this patch tested?

Existing tests.

bozhang2820 avatar Sep 19 '22 10:09 bozhang2820

@cloud-fan, could you take a look?

bozhang2820 avatar Sep 21 '22 01:09 bozhang2820

thanks, merging to master!

cloud-fan avatar Sep 22 '22 12:09 cloud-fan