[SPARK-44914][BUILD] Upgrade Apache Ivy to 2.5.2

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to upgrade Apache Ivy to 2.5.2.

Why are the changes needed?

This was tried once and reverted logically due to Java 11 and Java 17 failures in Daily CIs.

• #42613
• #42668

Since we dropped Java 11 and switched to Java 17 as the default CI, we can re-try for the following.

• Release notes
• https://github.com/advisories/GHSA-2jc4-r94c-rp7h
  • The fix https://github.com/apache/ant-ivy/commit/2be17bc18b0e1d4123007d579e43ba1a4b6fab3d

In addition, we backported the following to Apache Spark 3.5.1 and 3.4.3. This will mitigate the Ivy cache incompatibility issue.

• #45017

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs including HiveExternalCatalogVersionsSuite.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

Hmm. Although it passed locally, this seems to fail still in CI. Let me take a look.

[info] *** 1 SUITE ABORTED ***
[error] Error: Total 1555, Failed 0, Errors 1, Passed 1554, Ignored 597
[error] Error during tests:
[error] 	org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite

[LuciferYang]

Previously, my workaround solution was here: https://github.com/apache/spark/pull/44477/files

[dongjoon-hyun]

Ya, I'm digging differently with fresh eyes, @LuciferYang .

[dongjoon-hyun]

Could you review this, @LuciferYang ? Apache Spark 3.5.1 is accessible now. I believe we are ready for Ivy upgrade.

For further clean-ups, I'll proceed separate in SPARK-47126 (which is added as TODO).

[LuciferYang]

hmm... I'm not sure if SPARK-46400 can fix this issue at the same time. IIRC, these are two different issues. Let's wait for the test results from ci.

[dongjoon-hyun]

Oh, I didn't realize them. Do we have any JIRA issues? Then, I can track them together.

IIRC, these are two different issues.

[dongjoon-hyun]

To @LuciferYang , in any way, if we have more issues, our Maven CI will be broken from Today because we didn't protect them from 3.5.1. Let me make a PR for them while waiting this.

https://github.com/apache/spark/blob/a87015efb5cf36103bc4eb82ae8613874e2eb408/.github/workflows/build_maven.yml#L36

[LuciferYang]

I recorded the issue in the PR of revert SPARK-44914: https://github.com/apache/spark/pull/42668

and provided a manual reproduce method when attempting to upgrade again

https://github.com/apache/spark/pull/44477#issuecomment-1869925312

[info]   : java.lang.RuntimeException: problem during retrieve of org.apache.spark#spark-submit-parent-8bd00540-3ae3-45c0-b8cb-adf54c547a85: java.lang.RuntimeException: Multiple artifacts of the module log4j#log4j;1.2.17 are retrieved to the same file! Update the retrieve pattern to fix this error.
[info]    at org.apache.ivy.core.retrieve.RetrieveEngine.retrieve(RetrieveEngine.java:238)`

Previously, upgrading ivy to 2.5.2 would cause SBT testing to fail, not just Maven

[dongjoon-hyun]

Yes, I already verified SBT failures on this PR, @LuciferYang . That's the reason why this PR can be a way to verify Ivy issue before going with Daily Maven CI.

Previously, upgrading ivy to 2.5.2 would cause SBT testing to fail, not just Maven

[dongjoon-hyun]

IIUC, both Apache Spark 4.0.0-SNAPSHOT and 3.5.1 has the same patch and they should work together without Ivy 2.5.2 issue.

[dongjoon-hyun]

Anyway, let's wait and see as you told.

[dongjoon-hyun]

Too bad. It still fails.

[info] *** 1 SUITE ABORTED ***
[error] Error: Total 1553, Failed 0, Errors 1, Passed 1552, Ignored 597
[error] Error during tests:
[error] 	org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite

[LuciferYang]

hmm...

[dongjoon-hyun]

Could you review this again, @LuciferYang ?

[dongjoon-hyun]

Thank you, @LuciferYang and @srowen . Let me merge this first to see Daily Maven runs, too.