incubator-uniffle
incubator-uniffle copied to clipboard
Support lower Hadoop versions in client-mr
Currently, uniffle use the default Hadoop version of 2.8.5.
When using the ./build_distribution.sh --spark2-profile 'spark2' --spark3-mvn '-Dspark.version=2.4.3' --spark3-profile 'spark3' --spark3-mvn '-Dspark.version=3.1.1' -Dhadoop.version=2.6.0, it will throw exceptions due to some methods and vars not supported in Hadoop 2.6.0.
Some non-compatible params and methods as follows
CallContext, introduced by >= 2.8.0.MRJobConfig.DEFAULT_SHUFFLE_MERGE_PERCENTintroduced by 2.8.0. ticket linkMRApps.getSystemPropertiesToLogintroduced by 2.8.0 ticket link
I think we could use the reflection to be compatible with lower hadoop version.
cc @jerqi @frankliee
And I also found the client-mr is not compatible with Hadoop 3.2.2
What's your company's hadoop version?
- 2.6.0-cdh5.11.0
- Hadoop-3.2.2 packaged by bigtop
I think it's ok for me if we need it in our production environment.
OK. I will go ahead.