[SPARK-48867][BUILD] Upgrade `okhttp` to 4.12.0, `okio` to 3.9.0 and `esdk-obs-java` to 3.24.3

[roczei]

What changes were proposed in this pull request?

This PR aims to upgrade okhttp to 4.12.0, okio to 3.9.0 and esdk-obs-java to 3.24.3.

Why are the changes needed?

okhttp depends on okio which has to be upgraded as well. The new okhttp version fixes the following vulnerabilities:

CVE-2023-0833 - A flaw was found in Red Hat's AMQ-Streams, which ships a version of the OKHttp component with an information disclosure flaw via an exception triggered by a header containing an illegal value. This issue could allow an authenticated attacker to access information outside of their regular permissions.

CVSSv3 Score:- 5.5(Medium)

https://nvd.nist.gov/vuln/detail/CVE-2023-0833

CVE-2021-0341 - In verifyHostName of OkHostnameVerifier.java, there is a possible way to accept a certificate for the wrong domain due to improperly used crypto. This could lead to remote information disclosure with no additional execution privileges needed. User interaction is not needed for exploitation.

CVSSv3 Score:- 7.5(High)

https://nvd.nist.gov/vuln/detail/CVE-2021-0341 https://github.com/square/okhttp/issues/6724

There are two places in the Spark repository where the okhttp dependency comes in as transitive dependency:

[INFO] +- org.apache.hadoop:hadoop-cloud-storage:jar:3.4.0:compile [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile [INFO] | +- org.apache.hadoop:hadoop-aliyun:jar:3.4.0:compile [INFO] | | +- com.aliyun.oss:aliyun-sdk-oss:jar:3.13.2:compile [INFO] | | | +- org.jdom:jdom2:jar:2.0.6:compile [INFO] | | | +- com.aliyun:aliyun-java-sdk-core:jar:4.5.10:compile [INFO] | | | | +- org.ini4j:ini4j:jar:0.5.4:compile [INFO] | | | | +- io.opentracing:opentracing-api:jar:0.33.0:compile [INFO] | | | | - io.opentracing:opentracing-util:jar:0.33.0:compile [INFO] | | | | - io.opentracing:opentracing-noop:jar:0.33.0:compile [INFO] | | | +- com.aliyun:aliyun-java-sdk-ram:jar:3.1.0:compile [INFO] | | | - com.aliyun:aliyun-java-sdk-kms:jar:2.11.0:compile [INFO] | | - org.codehaus.jettison:jettison:jar:1.5.4:compile [INFO] | +- org.apache.hadoop:hadoop-azure-datalake:jar:3.4.0:compile [INFO] | | - com.microsoft.azure:azure-data-lake-store-sdk:jar:2.3.9:compile [INFO] | - org.apache.hadoop:hadoop-huaweicloud:jar:3.4.0:compile [INFO] | - com.huaweicloud:esdk-obs-java:jar:3.20.4.2:compile [INFO] | +- com.jamesmurty.utils:java-xmlbuilder:jar:1.2:compile [INFO] | +- com.squareup.okhttp3:okhttp:jar:3.14.2:compile [INFO] | - com.squareup.okio:okio:jar:1.17.6:compile

The Hadoop team has attempted to remove okhttp from their codebase:

remove okhttp usage: https://issues.apache.org/jira/browse/HADOOP-18890

Unfortunately the hadoop-huaweicloud dependency is still there which pulls in the vulnerable okhttp 3.x version.

https://github.com/apache/hadoop/blob/trunk/hadoop-cloud-storage-project/hadoop-cloud-storage/pom.xml#L137C19-L137C37

Proposed solution for this:

com.huaweicloud:esdk-obs-java:jar:3.20.4.2 is vulnerable due to okhttp 3.x (CVE-2023-0833, CVE-2021-0341), it has to be upgraded to 3.24.3 which depends on okhttp 4.12.0

[INFO] +- org.apache.spark:spark-kubernetes_2.13:jar:4.0.0-SNAPSHOT:compile [INFO] | +- io.fabric8:kubernetes-httpclient-okhttp:jar:6.13.3:compile [INFO] | | +- io.fabric8:kubernetes-client-api:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-core:jar:6.13.3:compile [INFO] | | | | - io.fabric8:kubernetes-model-common:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-gatewayapi:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-resource:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-rbac:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-admissionregistration:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-apps:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-autoscaling:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-apiextensions:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-batch:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-certificates:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-coordination:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-discovery:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-events:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-extensions:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-flowcontrol:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-networking:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-metrics:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-policy:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-scheduling:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-storageclass:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-node:jar:6.13.3:compile [INFO] | | | - org.snakeyaml:snakeyaml-engine:jar:2.7:compile [INFO] | | +- com.squareup.okhttp3:okhttp:jar:3.12.12:compile [INFO] | | | - com.squareup.okio:okio:jar:1.17.6:compile [INFO] | | - com.squareup.okhttp3:logging-interceptor:jar:3.12.12:compile

kubernet-client maintainers have decided to update okhttp from 3.x to 4.x in their upcoming version 7: https://github.com/fabric8io/kubernetes-client/issues/5778

My proposed solution based on the above finding:

Exclude the 3.x version and switch to use okhttp 4.x. Source: https://github.com/fabric8io/kubernetes-client/blob/main/doc/KubernetesClientWithIPv6Clusters.md

It is binary backwards compatible with okhttp 3.x. More details are here:

https://square.github.io/okhttp/upgrading_to_okhttp_4/

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs.

Was this patch authored or co-authored using generative AI tooling?

No.