[DSIP-89][JDK] Supports running on JDK17

[det101]

Search before asking

• [x] I had searched in the DSIP and found no similar DSIP.

Motivation

Long - Term Support (LTS): Java 17 is the latest LTS version, which means it will receive long - term support and updates. Performance Improvement: Java 17 introduces a variety of performance improvements, including optimizations in garbage collection and JIT compilation. New Features: Java 17 brings many new features, such as pattern matching, records, and sealed classes, all of which can make code more concise and readable. Enhanced Security: Java 8 has been discontinued, and Java 11 will no longer be updated after 2026. Security vulnerabilities will continue to increase. Upgrading to Java 17 can ensure the security of applications. Other Projects: The latest versions of many projects currently support JDK 17 and JDK 21, such as Spring, Spark, StreamPack, Doris, and so on.

Design Detail

Upgrade Plan DS has already supported JDK 11, so the upgrade to 1 should7 not be too difficult. We will proceed with the principle of forward compatibility and minimal code changes, with the ultimate goal of enabling the project to run on JDK 8, JDK 11, and JDK 17.

Before starting the upgrade, I will conduct the following research:

1. Evaluate static code compatibility issues using various tools such as jdeps, jdeprscan, and EMT4J to get a rough overview of the changes needed for the project. Discuss the results with the PMC.

Compatibility, Deprecation, and Migration Plan

No response

Test Plan

1. Compile with Java 8 and conduct integration tests in a Java 17 environment to identify and resolve issues.

  - All modifications are made with the principle of forward compatibility. If compatibility is not possible, provide clear explanations.
  - Conduct integration testing of the entire process to ensure that all tasks can be executed normally.

2. Verify whether E2E and UT tests can be conducted under JDK 17 and resolve any issues found.

3. Update community documentation.

Code of Conduct

• [x] I agree to follow this project's Code of Conduct

[det101]

[SbloodyS]

cc @Gallardot @zhongjiajie @ruanwenjun @caishunfeng @EricGao888

[zhongjiajie]

GLTM

[SbloodyS]

Thanks for bringing this. Before PR there are still some issues to be sorted out.

1. There are a lot of third-party cloud dependencies on data source types, task types and storage types, such as AWS, Tencent Cloud, Alibaba Cloud, GCP and so on. These closed-source tools can't be realized by open-source automated testing. How to ensure that these will work normally after the upgrade is completed?
2. Hadoop 2 does not support JDK17, and there are still a large number of users using Hadoop 2. How to achieve Hadoop 2 compatibility under JDK 17?
3. Many third-party open source dependencies need to be upgraded. How to ensure the compatibility after upgrading?

[det101]

Thanks for bringing this. Before PR there are still some issues to be sorted out.

1. There are a lot of third-party cloud dependencies on data source types, task types and storage types, such as AWS, Tencent Cloud, Alibaba Cloud, GCP and so on. These closed-source tools can't be realized by open-source automated testing. How to ensure that these will work normally after the upgrade is completed?
2. Hadoop 2 does not support JDK17, and there are still a large number of users using Hadoop 2. How to achieve Hadoop 2 compatibility under JDK 17?
3. Many third-party open source dependencies need to be upgraded. How to ensure the compatibility after upgrading?

The purpose of the upgrade is to enable DS to run on JDK17. The problem you mentioned can be solved by the following methods:

1. Regarding third-party cloud dependencies, including compatibility issues of data source types, task types, and storage types, I would first read the developer documentation of all third-party clouds to understand the support of the current reference version of DS for JDK17. If it is clearly not supported, check whether the new version of the dependency is forward compatible with lower-version services (I have seen S3 1.12 versions support JDK17). If there is a situation where a higher-version dependency cannot support a lower-version cloud service, then users can only choose to run on JDK8/11.

2. There should be no problem submitting through the command line (such as hive -e, spark-sql, hdfs commands). Components use their own dependent JDKs, and DS uses JDK17 (multiple JDK environments are also common now). Similar to hadoop dependencies, low-version JDK dependencies may indeed have problems under JDK17 (to be verified). This issue requires determining which version hadoop DS supports by default. Or if it is 2.x, DS at least provides support for JDK17, and users only need to update the hadoop3-related dependency version. Regarding (hadoop3+DS), I can fully verify it in JDK17.

[SbloodyS]

If there is a situation where a higher-version dependency cannot support a lower-version cloud service, then users can only choose to run on JDK8/11.

This needs to be discussed. @zhongjiajie @ruanwenjun @Gallardot @caishunfeng

There should be no problem submitting through the command line (such as hive -e, spark-sql, hdfs commands).

Yes. Command line operation will not be affected. What I mean is many dependencies in pom.xml don't support jdk17. If you upgrade, you need to change the code. How to ensure the compatibility after upgrading? We need full design detail of all dependencies that need to be upgraded.

[det101]

If there is a situation where a higher-version dependency cannot support a lower-version cloud service, then users can only choose to run on JDK8/11.

This needs to be discussed. @zhongjiajie @ruanwenjun @Gallardot @caishunfeng

There should be no problem submitting through the command line (such as hive -e, spark-sql, hdfs commands).

Yes. Command line operation will not be affected. What I mean is many dependencies in pom.xml don't support jdk17. If you upgrade, you need to change the code. How to ensure the compatibility after upgrading? We need full design detail of all dependencies that need to be upgraded.

I will try it in our environment first to see what problems I will encounter, and will share the problems with the community for your reference

[ruanwenjun]

If there is a situation where a higher-version dependency cannot support a lower-version cloud service, then users can only choose to run on JDK8/11.

This needs to be discussed. @zhongjiajie @ruanwenjun @Gallardot @caishunfeng

There should be no problem submitting through the command line (such as hive -e, spark-sql, hdfs commands).

Yes. Command line operation will not be affected. What I mean is many dependencies in pom.xml don't support jdk17. If you upgrade, you need to change the code. How to ensure the compatibility after upgrading? We need full design detail of all dependencies that need to be upgraded.

AFAIK, JDK17 is compatible with previous version, e.g. 8, 11, maybe some JVM args are not compatible, this is acceptable since we only provide the basic jvm args in the source code. But this still need to test, the ci can only make sure it can compile/run the basic case in JDK17.

[det101]

If there is a situation where a higher-version dependency cannot support a lower-version cloud service, then users can only choose to run on JDK8/11.

This needs to be discussed. @zhongjiajie @ruanwenjun @Gallardot @caishunfeng

There should be no problem submitting through the command line (such as hive -e, spark-sql, hdfs commands).

Yes. Command line operation will not be affected. What I mean is many dependencies in pom.xml don't support jdk17. If you upgrade, you need to change the code. How to ensure the compatibility after upgrading? We need full design detail of all dependencies that need to be upgraded.

AFAIK, JDK17 is compatible with previous version, e.g. 8, 11, maybe some JVM args are not compatible, this is acceptable since we only provide the basic jvm args in the source code. But this still need to test, the ci can only make sure it can compile/run the basic case in JDK17.

Yes, for incompatible JVM parameters, we need to provide different JVM configurations, or start checking the version to select the best configuration. Of course, this requires system testing.

[ruanwenjun]

If there is a situation where a higher-version dependency cannot support a lower-version cloud service, then users can only choose to run on JDK8/11.

This needs to be discussed. @zhongjiajie @ruanwenjun @Gallardot @caishunfeng

There should be no problem submitting through the command line (such as hive -e, spark-sql, hdfs commands).

Yes. Command line operation will not be affected. What I mean is many dependencies in pom.xml don't support jdk17. If you upgrade, you need to change the code. How to ensure the compatibility after upgrading? We need full design detail of all dependencies that need to be upgraded.

AFAIK, JDK17 is compatible with previous version, e.g. 8, 11, maybe some JVM args are not compatible, this is acceptable since we only provide the basic jvm args in the source code. But this still need to test, the ci can only make sure it can compile/run the basic case in JDK17.

I find hadoop2 sdk cannot use JDK17 due to some reflect usage.

[det101]

If there is a situation where a higher-version dependency cannot support a lower-version cloud service, then users can only choose to run on JDK8/11.

This needs to be discussed. @zhongjiajie @ruanwenjun @Gallardot @caishunfeng

There should be no problem submitting through the command line (such as hive -e, spark-sql, hdfs commands).

Yes. Command line operation will not be affected. What I mean is many dependencies in pom.xml don't support jdk17. If you upgrade, you need to change the code. How to ensure the compatibility after upgrading? We need full design detail of all dependencies that need to be upgraded.

AFAIK, JDK17 is compatible with previous version, e.g. 8, 11, maybe some JVM args are not compatible, this is acceptable since we only provide the basic jvm args in the source code. But this still need to test, the ci can only make sure it can compile/run the basic case in JDK17.

I find hadoop2 sdk cannot use JDK17 due to some reflect usage.

Is it possible to use compilation parameters to support hadoop 2x_java8_11 and hadoop3x_java8_11_17? By the way, Hadoop will support jdk17 in 3.5 https://issues.apache.org/jira/browse/HADOOP-17177

[SbloodyS]

Is it possible to use compilation parameters to support hadoop 2x_java8_11 and hadoop3x_java8_11_17? By the way, Hadoop will support jdk17 in 3.5 https://issues.apache.org/jira/browse/HADOOP-17177

I can't estimate the scope of influence since hadoop2 may not be the only problem. If the scope is too large, I think the upgrade is not worth the candle.