hudi icon indicating copy to clipboard operation
hudi copied to clipboard
verified publisher icon apache

[WIP][HUDI-7915] Support Spark 4.0.0

Open CTTY opened this issue 1 year ago • 5 comments

Change Logs

Support Spark 4.0.0

Impact

No public-facing API changes

Risk level (write none, low medium or high below)

Medium

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

  • The config description must be updated if new configs are added or the default value of the configs are changed
  • Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the ticket number here and follow the instruction to make changes to the website.

Contributor's checklist

  • [ ] Read through contributor's guide
  • [ ] Change Logs and Impact were stated clearly
  • [ ] Adequate tests were added if applicable
  • [ ] CI passed

CTTY avatar Jun 26 '24 04:06 CTTY

@CTTY Are you planning to apply this to Amazon EMR Hudi once merged?

Gatsby-Lee avatar Jun 26 '24 18:06 Gatsby-Lee

@CTTY Are you planning to apply this to Amazon EMR Hudi once merged?

@Gatsby-Lee Yes, the support would be released along with Spark 4.0.0 on EMR

CTTY avatar Jun 28 '24 22:06 CTTY

@CTTY any specific target EMR release version with Spark 4.0 support?

Gatsby-Lee avatar Jun 28 '24 22:06 Gatsby-Lee

@Gatsby-Lee No, but we should have better idea after Spark 4 offcial release is out

CTTY avatar Jun 28 '24 23:06 CTTY

@CTTY got it. thank you!!

Gatsby-Lee avatar Jun 28 '24 23:06 Gatsby-Lee

Thank you for contribution, but as this PR https://github.com/apache/hudi/pull/11539 is not updated more than half a year, I pulled the code from there, rebased on actual master, resolved conflicts and compilation issues, fixed tests. If you don't mind, I close this.

Actual PR is here: https://github.com/apache/hudi/pull/12772

wombatu-kun avatar Feb 13 '25 09:02 wombatu-kun