[HUDI-7915] Spark4 + Hadoop3

[CTTY]

Change Logs

Support Spark 4.0.0 + Hadoop 3.4

Impact

No public-facing API changes

Risk level (write none, low medium or high below)

Medium

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the ticket number here and follow the instruction to make changes to the website.

Contributor's checklist

• [ ] Read through contributor's guide
• [ ] Change Logs and Impact were stated clearly
• [ ] Adequate tests were added if applicable
• [ ] CI passed

[CTTY]

Opened this PR for testing hadoop 3.4 on top of Spark4. When testing Spark4 I saw many tests failed to java.lang.NoSuchMethodError: 'boolean org.apache.hadoop.fs.FileStatus.hasAcl()'. FileStatus#hasAcl only exists in Hadoop 3+

[CTTY]

This PR is still WIP, I'm trying to trigger Azure CI to detect potential failures

[CTTY]

@hudi-bot run azure

[hudi-bot]

CI report:

• dac29c7e89201f0ced6d394bf6fd4a5c0622167b UNKNOWN
• 89fa17eba4799522f7b6617a957f844f6b72b72a UNKNOWN
• 7ddb39305607086421df337239ae1cd9fe800382 UNKNOWN
• b878d07d4ae53a63bc5dadf04adffbff23b499c1 UNKNOWN
• 842b541686710ff43e799913c8888b72a34da07f Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[yihua]

Closing this in favor of #12772