datafusion-comet
datafusion-comet copied to clipboard
apache
fix: Support scalar/array args for rpad/read_side_padding
Which issue does this PR close?
Closes #2475.
Rationale for this change
Support full scalar/array arguments for rpad/read_side_padding
What changes are included in this PR?
Use make_scalar_function to convert arguments to arrays.
How are these changes tested?
unit test
Codecov Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 58.40%. Comparing base (f09f8af) to head (d57ea64).
:warning: Report is 560 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@
## main #2482 +/- ##
============================================
+ Coverage 56.12% 58.40% +2.28%
- Complexity 976 1436 +460
============================================
Files 119 146 +27
Lines 11743 13510 +1767
Branches 2251 2350 +99
============================================
+ Hits 6591 7891 +1300
- Misses 4012 4389 +377
- Partials 1140 1230 +90
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
Failed spark test case:
[info] - SPARK-48498: always do char padding in predicates *** FAILED *** (330 milliseconds)
[info] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 605.0 failed 1 times, most recent failure: Lost task 0.0 in stage 605.0 (TID 582) (735d43d611a7 executor driver): org.apache.comet.CometNativeException: Unsupported data type (Dictionary(Int32, Utf8), Utf8) for function rpad/read_side_padding.
[info] This issue was likely caused by a bug in DataFusion's code. Please help us to resolve this by filing a bug report in our issue tracker: https://github.com/apache/datafusion/issues
[info] at org.apache.comet.Native.executePlan(Native Method)
@coderfender as were working on rpad/lpad
Thank you for the info, feel free to take over this.
@wForget , If my understanding is right, you are adding support to non scalar args to rpad function is it ? I can take a stab at the PR this weekend , fix the merge conflicts and make changes necessary to match spark's behavior and I hope that is Ok ?
