support Spark3.2 and Java11

[a140262]

What would you like to be added?

I am trying to build a CSS docker image to support Spark3.2 based on java11 (FROM amazoncorretto:11 ), not able to compile it so far. How can I make it work with Spark3.2? It is acceptable if CSS works with Spark3.2 & java8. Please help.

Here is the list of version changes in pom.xml file:

<java.version>11</java.version>
<maven.compiler.source>${java.version}</maven.compiler.source>
<maven.compiler.target>${java.version}</maven.compiler.target>
<maven.version>3.8.6</maven.version>
<spark.version>3.2.0</spark.version>
<hadoop.version>3.2.1</hadoop.version>

Error:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project css-common_2.12: Compilation failure: Compilation failure: 
[ERROR] /tmp/CloudShuffleService/common/src/main/java/com/bytedance/css/common/unsafe/Platform.java:[27,16] cannot find symbol
[ERROR]   symbol:   class Cleaner
[ERROR]   location: package sun.misc
[ERROR] /tmp/CloudShuffleService/common/src/main/java/com/bytedance/css/common/unsafe/Platform.java:[174,7] cannot find symbol
[ERROR]   symbol:   class Cleaner
[ERROR]   location: class com.bytedance.css.common.unsafe.Platform
[ERROR] /tmp/CloudShuffleService/common/src/main/java/com/bytedance/css/common/unsafe/Platform.java:[174,25] cannot find symbol
[ERROR]   symbol:   variable Cleaner
[ERROR]   location: class com.bytedance.css.common.unsafe.Platform
[ERROR] -> [Help 1]

The replication step is:

1. spin up a jdk11 container with maven installed:

docker build -t jdk11 .
docker run -it jdk11

The jdk11 Dockerfile looks like this:

FROM amazoncorretto:11
ARG MAVEN_VERSION=3.8.6
ARG BASE_URL=https://apache.osuosl.org/maven/maven-3/${MAVEN_VERSION}/binaries

# install maven
RUN yum update -y && yum install -y git tar
RUN mkdir -p /usr/share/maven /usr/share/maven/ref \
 && curl -fsSL -o /tmp/apache-maven.tar.gz ${BASE_URL}/apache-maven-${MAVEN_VERSION}-bin.tar.gz \
 && tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven --strip-components=1 \
 && rm -f /tmp/apache-maven.tar.gz \
 && ln -s /usr/share/maven/bin/mvn /usr/bin/mvn
ENV MAVEN_HOME /usr/share/maven
ENV MAVEN_CONFIG "$USER_HOME_DIR/.m2"

2. After login to the jdk11 container, upgrade spark & java versions in pom.xml, then compile the project:

git clone https://github.com/bytedance/CloudShuffleService.git /tmp/CloudShuffleService
cd /tmp/CloudShuffleService
vi pom.xml  
./build.sh

Why is this needed?

need to run the latest Spark with the CSS.

[bdyx123]

We support Spark 3.0 currently, and we are doing the support for Spark 3.2

[bdyx123]

We have tested it, spark-shuffle-manager-3 is also suitable for Spark 3.2 @a140262

[melodyyangaws]

how to compile it for Spark 3.2? these didn't work for me when compiling the CSS .

<java.version>11</java.version>
<spark.version>3.2.0</spark.version>
<hadoop.version>3.2.1</hadoop.version> 

Please provide more details on your test.