Spark benchmarks broken by JDK-8296244

[dougxc]

As a result of JDK-8296244 some/all Spark benchmarks in renaissance are broken:

Benchmark 'als' failed with exception:
java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed
	at java.base/javax.security.auth.Subject.getSubject(Subject.java:347)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:577)
	at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2416)
	at scala.Option.getOrElse(Option.scala:201)
	at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2416)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:334)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2888)
	at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1099)
	at scala.Option.getOrElse(Option.scala:201)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1093)
	at org.renaissance.apache.spark.SparkUtil.setUpSparkContext(SparkUtil.scala:81)
	at org.renaissance.apache.spark.SparkUtil.setUpSparkContext$(SparkUtil.scala:54)
	at org.renaissance.apache.spark.Als.setUpSparkContext(Als.scala:43)
	at org.renaissance.apache.spark.Als.setUpBeforeAll(Als.scala:88)
	at org.renaissance.harness.ExecutionDriver.executeBenchmark(ExecutionDriver.java:82)
	at org.renaissance.harness.RenaissanceSuite$.runBenchmarks$$anonfun$1(RenaissanceSuite.scala:172)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at scala.collection.immutable.List.foreach(List.scala:333)
	at org.renaissance.harness.RenaissanceSuite$.runBenchmarks(RenaissanceSuite.scala:205)
	at org.renaissance.harness.RenaissanceSuite$.main(RenaissanceSuite.scala:130)
	at org.renaissance.harness.RenaissanceSuite.main(RenaissanceSuite.scala)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.renaissance.core.Launcher.loadAndInvokeHarnessClass(Launcher.java:129)
	at org.renaissance.core.Launcher.launchHarnessClass(Launcher.java:78)
	at org.renaissance.core.Launcher.main(Launcher.java:43)

As far as I can see, any app that does not explicitly set the java.security.manager system property can no longer call javax.security.auth.Subject.getSubject().

[lbulej]

Thanks for the report! To be completely honest, the JDK bug info you reference does not make any sense to me (I never really needed to fiddle with security manager and I was under impression that security manager is actually going away, not that it's going to cause more problems :-) Is there some transition guide (or a reasonable default setting for the java.security.manager property) on how to unbreak apps/frameworks that JDK cleanups/enhancements keep breaking (Spark in particular)?

[dougxc]

I'm not aware of any particular guide apart from deprecation warnings that more recent versions of javac must be emitting when compiling Spark code that used deprecated API.

It might be worth encouraging Spark maintainers to periodically look at these warnings and plan for addressing them.

[wangweij]

I'm from Oracle's Java Platform Group. Many thanks for trying out the jdk23 early builds and reporting this issue. The Subject::getSubject method takes an AccessControlContext argument and it will be non-functional after Security Manager is removed. JDK-8296244 is meant to provide an alternative implementation so users can experience it right now. Please read the CSR and the Release Note(not delivered yet) for more details. You can try the Subject::current method and see if it works for you, or, as a workaround, add -Djava.security.manager=allow to keep the old bahavior.

[AlanBateman]

Just to add to Weijun's comment. Subject.getSubject(ACC) has been deprecated for removal since Java 17. Developers are encouraged to move to the replacement Subject.current(). I think we should make sure there is an issue in the Hadoop issue tracker for this.

[lbulej]

The Subject::getSubject method takes an AccessControlContext argument and it will be non-functional after Security Manager is removed. JDK-8296244 is meant to provide an alternative implementation so users can experience it right now. Please read the CSR and the Release Note(not delivered yet) for more details.

Thanks for the links!

You can try the Subject::current method and see if it works for you, or, as a workaround, add -Djava.security.manager=allow to keep the old behavior.

Great! This might be just what's needed for Renaissance. I'll check if we can setup the right environment for Spark in the harness so that its calls to Hadoop do not fail on this.