[ZEPPELIN-5969] Remove Hadoop2 Support

[Reamer]

What is this PR for?

This pull request removes support for Hadoop 2.

What type of PR is it?

Improvement

Todos

• [ ] - Fix CI
• [ ] - Check License files

What is the Jira issue?

• https://issues.apache.org/jira/browse/ZEPPELIN-5969

How should this be tested?

• CI

Questions:

• Does the license files need to update? Yes
• Is there breaking changes for older versions? Yes
• Does this needs documentation? Yes

[pan3793]

Removing Hadoop2 support would be a big change, I'm not very familiar with Flink, but for Spark, it requires at least Spark 3.x to support Hadoop 3, the Livy module still uses Spark 2.4.8.

And Spark moved to use Hadoop Shaded client since 3.2.0, which is a quite big change for Hadoop deps management.

I suppose it requires much effort to complete this task, as the community is preparing for the 0.11 release, better to postpone this change to the next release.

[Reamer]

Closed in favour of #4691