[HUDI-8089] Remove support for Spark2 and Scala2.11 from Hudi

[jonvex]

Change Logs

Remove Spark2 and Scala2.11 from Hudi. Try to clean up simple things. Put todo's for potentially more complex cleanup for HUDI-8086.

Impact

Don't need to support spark2 and scala2.11 anymore. Less work to do when making changes that have issues between versions. Can use newer features of spark and scala.

Risk level (write none, low medium or high below)

medium

Documentation Update

N/A

Contributor's checklist

• [ ] Read through contributor's guide
• [ ] Change Logs and Impact were stated clearly
• [ ] Adequate tests were added if applicable
• [ ] CI passed

[hudi-bot]

CI report:

• 607e9d3e2b7fe912c9b6e4bb64dae1e0f2378ed9 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build