Upgrade Hadoop to 3.3.4

[zhangbutao]

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

[sonarqubecloud[bot]]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
No Duplication information

[zhangbutao]

the build will pass, I tried till 3.3.3 when trying to upgrade, hive doesn't pose any issues there, but Tez doesn't work with Hive in actual cluster that is why we settled to 3.3.1. You need to upgrade Tez as well to 3.3.4 and then upgrade Hive to that Tez version. Can check more details on the previous 3.3.1 upgrade PR...

@ayushtkn Thanks for the info. Maybe i should test this PR with actual cluster and i will check this. I also foud the Tez ticket about upgrading hadoop to 3.3.3 https://github.com/apache/tez/pull/213. If I understand you correctly, we should deal with both tickets at the same time. As hadoop-3.3.4 is a latest stable version, updating hadoop from 3.3.1 to 3.3.4 is a good choice.

[ayushtkn]

Yes. you got me right, we should do both the tickets

[zhangbutao]

Tested this Pr with Hadoop3.3.4 & Tez0.10.2, and HS2 started failed with above WARN. This is because Tez0.10.2 added related hadoop3.3.1 jars(e.g. hadoop-hdfs-client-3.3.1.jar) to binary package and this caused some conflict in hive's classpath.

If i change the hadoop version to 3.3.4 in Tez0.10.2, hs2 will run successfully. :)

[github-actions[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the [email protected] list if the patch is in need of reviews.