incubator-uniffle icon indicating copy to clipboard operation
incubator-uniffle copied to clipboard

Support lower Hadoop versions in client-mr

Open zuston opened this issue 3 years ago • 6 comments

Currently, uniffle use the default Hadoop version of 2.8.5.

When using the ./build_distribution.sh --spark2-profile 'spark2' --spark3-mvn '-Dspark.version=2.4.3' --spark3-profile 'spark3' --spark3-mvn '-Dspark.version=3.1.1' -Dhadoop.version=2.6.0, it will throw exceptions due to some methods and vars not supported in Hadoop 2.6.0.

Some non-compatible params and methods as follows

  1. CallContext, introduced by >= 2.8.0.
  2. MRJobConfig.DEFAULT_SHUFFLE_MERGE_PERCENT introduced by 2.8.0. ticket link
  3. MRApps.getSystemPropertiesToLog introduced by 2.8.0 ticket link

I think we could use the reflection to be compatible with lower hadoop version.

zuston avatar Aug 18 '22 05:08 zuston

cc @jerqi @frankliee

zuston avatar Aug 18 '22 05:08 zuston

And I also found the client-mr is not compatible with Hadoop 3.2.2

zuston avatar Aug 18 '22 05:08 zuston

What's your company's hadoop version?

jerqi avatar Aug 18 '22 07:08 jerqi

  1. 2.6.0-cdh5.11.0
  2. Hadoop-3.2.2 packaged by bigtop

zuston avatar Aug 18 '22 07:08 zuston

I think it's ok for me if we need it in our production environment.

jerqi avatar Aug 18 '22 07:08 jerqi

OK. I will go ahead.

zuston avatar Aug 18 '22 10:08 zuston