spark icon indicating copy to clipboard operation
spark copied to clipboard

[SPARK-47977] DateTimeUtils.timestampDiff and DateTimeUtils.timestampAdd should not throw INTERNAL_ERROR exception

Open vitaliili-db opened this issue 1 year ago • 2 comments

What changes were proposed in this pull request?

Convert INTERNAL_ERROR for timestampAdd and timestampDiff to error with class. Reusing INVALID_PARAMETER_VALUE.DATETIME_UNIT used when parsing expressions.

The change is needed since timestampDiff and timestampAdd expressions could be constructed without going through parser - e.g. PySpark creates timestampDiff through PythonSQLUtils.

Why are the changes needed?

Correct classification of errors

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing unit tests updated with new error class

Was this patch authored or co-authored using generative AI tooling?

No

vitaliili-db avatar Apr 24 '24 21:04 vitaliili-db

@MaxGekk please take a look

vitaliili-db avatar Apr 24 '24 21:04 vitaliili-db

@cloud-fan please review

vitaliili-db avatar Apr 30 '24 17:04 vitaliili-db

it has merge conflicts, @vitaliili-db can you fix it?

cloud-fan avatar May 31 '24 00:05 cloud-fan

@cloud-fan done!

vitaliili-db avatar May 31 '24 17:05 vitaliili-db

Thanks, merging to master

gengliangwang avatar Jun 03 '24 18:06 gengliangwang