YangJie

cc @SethTisue

I don't have any new ideas at present

No better idea, close it first

> We intend to fallback to [Spark-3](https://issues.apache.org/jira/browse/SPARK-3).3.1 Scala-2.12 (instead of Scala 2.13) @tamama Using Scala 2.12 can avoid this issue?

> > > We intend to fallback to [Spark-3](https://issues.apache.org/jira/browse/SPARK-3).3.1 Scala-2.12 (instead of Scala 2.13) > > > > > > @tamama Using Scala 2.12 can avoid this issue? > >...

Merged into master for Spark 4.0. Thanks @pan3793 @srowen @tgravescs @mridulm

Previously, my workaround solution was here: https://github.com/apache/spark/pull/44477/files 

hmm... I'm not sure if SPARK-46400 can fix this issue at the same time. IIRC, these are two different issues. Let's wait for the test results from ci.

I recorded the issue in the PR of revert SPARK-44914: https://github.com/apache/spark/pull/42668 and provided a manual reproduce method when attempting to upgrade again https://github.com/apache/spark/pull/44477#issuecomment-1869925312  ``` [info] : java.lang.RuntimeException: problem during...

hmm...