BIGTOP-3852: Java 11 to bigtop-toolchain to use it in docker container during packaging

[yoda-mon]

Description of PR

This PR aims bigtop users can use Java 11 when they package some projects on a docker container.

• The behavior of bigtop is not changed.
• If user set an option, bigtop pass a flag to the docker container by environment variable. If build target project can receive the flag, bigtop switch JAVA_HOME to change the Java version.

How was this patch tested?

Installation of JDK 11

I checked below on aarch 64 machine

# build bigtop/slave
cd docker/bigtop-slaves/
OS_LIST=( ubuntu-18.04 ubuntu-20.04 ubuntu-22.04 debian-10 debian-11 centos-7 rockylinux-8 fedora-35 fedora-36 )
for OS in ${OS_LIST[@]} ; do
  ./build.sh trunk-${OS}
done

# check the installation
for OS in ${OS_LIST[@]} ; do
  echo $OS; 
  docker run -i --rm bigtop/slaves:trunk-$OS-aarch64 ls /usr/lib/jvm;
  echo ""
done

The default JAVA_HOME is set to 8 by /etc/profile.d/bigtop.sh so installation does not affect on it.

Build Kafka 2.8.1 by Java 11

I checked below on aarch 64 machine.

./gradlew zookeeper-pkg-ind kafka-pkg-ind -Ppreferred-java-version=11 -POS=rockylinux-8

I checked Java version by inserting mvn --version or gradle --version to do-component-build. Zookeeper was build by 8 and Kafka was by 11.

For code changes:

• [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'BIGTOP-3638. Your PR title ...')?
• [x] Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/

[yoda-mon]

I chose the way that has small impact, I think there are other ways to tell the selection of JDK to the container (like -P{project}-java-version=... and swich JAVA_HOME by groovy script), I would like to ask other's opinion.

[iwasakims]

I chose the way that has small impact, I think there are other ways to tell the selection of JDK to the container (like -P{project}-java-version=... and swich JAVA_HOME by groovy script), I would like to ask other's opinion.

We can not yet use Java 11 for building all products. Tweaking Java version in do-component-build for specific product should be fine now.

[guyuqi]

@yoda-mon Thanks for working on it. As far as I know, JDK 11 hasn't been totally supported in Apache world (some components don't support JDK11). IMHO, it is better to retain the JDK8 and make JDK8 the default jvm/jdk in Bigtop, and JDK11 could be installed the other separate path as the optional tool for users. :-)

[yoda-mon]

@iwasakims

Tweaking Java version in do-component-build for specific product should be fine now.

Yes, I think so too. This way does not harm other projects.

@guyuqi

it is better to retain the JDK8 and make JDK8 the default jvm/jdk in Bigtop, and JDK11 could be installed the other separate path as the optional tool for users.

I agree. The main JDK on Bigtop should be 8 because some project don't have compatibly for >8. However it is tough thing that several projects seem deprecating to use Java 8 on current versions, such as Kafka or Flink... To catch up them, we have to look around balanced way.

[yoda-mon]

I added several procedures to ensure to use java 8, not java 11.

• Add dependency to the class between jdk.pp and jdk11.pp for ordering. packaging systems uses alternatives for java, and later installed version has higher priority to be default (cf https://linux.die.net/man/8/update-alternatives).
• Ordering is not enough for some OSs, in this case, Debian/Ubuntu and Fedora 35, because other libraries require java 8 before installing java 11. So I ensure to use java 8 by using update-java-alternatives or update-alternatives. We can use update-java-alternatives on java that from apt packaging, it is simple. We can only use update-alternatives on java that RHEL based OS's packaging system, we have to set both java and javac (Other commands are hierarchically controlled under these two, you can check by update-alternatives --display java.

Here shows the script for a checking.

OS_LIST=(centos-7 rockylinux-8 fedora-35 fedora-36 debian-10 debian-11 ubuntu-18.04 ubuntu-20.04 ubuntu-22.04)
for OS in "${OS_LIST[@]}"
do
  echo $OS
  docker run --rm -itd --name bigtop-test-${OS} bigtop/slaves:trunk-${OS}-aarch64 /bin/bash
  docker exec -it bigtop-test-${OS} java -version
  docker exec -it bigtop-test-${OS} javac -version
  docker exec -it bigtop-test-${OS} bash -cl 'echo "$JAVA_HOME"'
  docker stop bigtop-test-${OS}
  echo ""
done

[yoda-mon]

I modified to use uname. In addition, I found dpkg --print-architecture shows arm64 or amd64 which is used as suffix of the java alternatives. So I add it and checked it worked on both CPU architectures.

[timyuer]

@yoda-mon Sorry to bother you, I found that there is already a BIGTOP_JDK variable in the bigtop.bom file that can switch JDK versions. Why do we need to create a new variable called preferred-java-version here ?

@iwasakims @guyuqi Can help take a look together?

[yoda-mon]

@timyuer Several years ago some packages do not compatible with Java 11 (For example, Hadoop only supported running on 11 and did not support building with 11) . However, some pakages started to omit suporting 8. I intended to mitigate these situations with adding another options here(I think these situations have been getting well).

[timyuer]

@timyuer Several years ago some packages do not compatible with Java 11 (For example, Hadoop only supported running on 11 and did not support building with 11) . However, some pakages started to omit suporting 8. I intended to mitigate these situations with adding another options here(I think these situations have been getting well).

@yoda-mon I agree that most components of BigTop have started or completed supporting JDK17, so I think this scenario is unnecessary. Moreover, adding a separate JDK option can easily cause ambiguity. I suggest that we use the BIGTOP_JDK variable to uniformly control the JDK version. What do you think?

[timyuer]

@timyuer Several years ago some packages do not compatible with Java 11 (For example, Hadoop only supported running on 11 and did not support building with 11) . However, some pakages started to omit suporting 8. I intended to mitigate these situations with adding another options here(I think these situations have been getting well).

In fact, using the BIGTOP_JDK variable can also solve the above problems.

[yoda-mon]

@timyuer

I agree that most components of BigTop have started or completed supporting JDK17, so I think this scenario is unnecessary.

I think Hadoop would be the blocker, wouldn't it ? https://issues.apache.org/jira/browse/HADOOP-17177

using the BIGTOP_JDK variable can also solve the above problems.

I think this is partially correct. If you build multi packages with

• same JDK at the same time or
• different JDK separatelly it's enough only to use BIGTOP_JDK However,
• different JDK at the same time it's not enough.

[timyuer]

@timyuer

I agree that most components of BigTop have started or completed supporting JDK17, so I think this scenario is unnecessary.

I think Hadoop would be the blocker, wouldn't it ? https://issues.apache.org/jira/browse/HADOOP-17177

using the BIGTOP_JDK variable can also solve the above problems.

I think this is partially correct. If you build multi packages with

• same JDK at the same time or
• different JDK separatelly it's enough only to use BIGTOP_JDK However,
• different JDK at the same time it's not enough.

In the third point, using different jdk at the same time can be subdivided into the following two types,

• Using two different versions of JDK at the same time, BIGTOP_JDK and preferred-java-version can be satisfied.
• Using three or more different versions of JDK at the same time, neither of the above two can be satisfied.

[timyuer]

This leads to the following discussion:

1. Is it necessary to use two different JDK compilers at the same time? Will this complicate deployment?
2. If supported, should it also support three or more different JDK compilations at the same time?