spark icon indicating copy to clipboard operation
spark copied to clipboard

[SPARK-39091][SQL] Updating specific SQL Expression traits that don't compose when multiple are extended due to nodePatterns being final.

Open neilagupta opened this issue 3 years ago • 6 comments

When creating classes that extend multiple Expression traits, they have an inheritance conflict due to nodePatterns being different values. nodePatterns is also final, which makes the inheritance conflict unavoidable.

This error comes in after trying to override the field - value nodePatterns in trait TimeZoneAwareExpression of type Seq[org.apache.spark.sql.catalyst.trees.TreePattern.TreePattern] cannot override final member.

What changes were proposed in this pull request?

For now kept the traits changed to a minimum - removed nodePatterns final classifier from TimeZoneAwareExpression, HigherOrderFunctions, and Generator. I have propagated this change to more children classes of Expression, but this change can be expanded (or unexpanded) to other traits if needed.

Why are the changes needed?

Enables multiple trait inheritance for certain Expression types. Hand-picked a few of them to have their nodePatterns attribute not final.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added a test that creates a dummy class that inherits all three of TimeZoneAwareExpression, HigherOrderFunctions, and Generator - the test is only to see if this can compile now. Since it compiles and runs, this issue is resolved for those three traits at minimum.

neilagupta avatar May 03 '22 16:05 neilagupta

Can one of the admins verify this patch?

AmplabJenkins avatar May 04 '22 02:05 AmplabJenkins

This change would be valuable for me. I imagine that if manually overidden, one would have to specify all bit patterns used in order to get all appropriate rules applied.

Seq(TIME_ZONE_AWARE_EXPRESSION, HIGHER_ORDER_FUNCTION, GENERATOR)

is that correct?

HuwCampbell avatar May 09 '22 06:05 HuwCampbell

Hi @HuwCampbell that should be correct, I will update this PR with more modified subclasses of Expression to have their nodePatterns fields not final (and to fix styling issues). Would that be in your interest?

neilagupta avatar May 12 '22 04:05 neilagupta

Yes indeed, this would unblock me from upgrading to Spark 3.2 once there's a release.

HuwCampbell avatar May 12 '22 04:05 HuwCampbell

@AmplabJenkins any chance I could get someone with write access to review this?

neilagupta avatar May 17 '22 16:05 neilagupta

@sigmod wanna take a look?

hvanhovell avatar Jul 14 '22 20:07 hvanhovell

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

github-actions[bot] avatar Mar 01 '23 00:03 github-actions[bot]