[SUPPORT] EMR on EKS version 6.15.0, Spark 3.4.1 and Hudi 0.14.0 getting java.io.IOException: Failed to delete: /usr/lib/hudi/.

[Lakshmi-Holla12]

Describe the problem you faced

Getting Failed to delete the hudi jar from /usr/lib/hudi/./ location after upgrading the EMR on EKS to 6.15.0 which has in built hudi 0.14.0 and Spark 3.4.1 version.

The application fails at the start of the spark application with the below error message,

Files  local:///usr/lib/hudi/hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar from /usr/lib/hudi/hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar to /usr/lib/hudi/./hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar
23/12/19 10:11:38 WARN JavaUtils: Attempt to delete using native Unix OS command failed for path = /usr/lib/hudi/./hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar. Falling back to Java IO way
java.io.IOException: Failed to delete: /usr/lib/hudi/./hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar
	at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingUnixNative(JavaUtils.java:177) ~[spark-network-common_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:113) ~[spark-network-common_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:94) ~[spark-network-common_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1232) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$14(SparkSubmit.scala:436) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286) ~[scala-library-2.12.15.jar:?]
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) ~[scala-library-2.12.15.jar:?]
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) ~[scala-library-2.12.15.jar:?]
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) ~[scala-library-2.12.15.jar:?]
	at scala.collection.TraversableLike.map(TraversableLike.scala:286) ~[scala-library-2.12.15.jar:?]
	at scala.collection.TraversableLike.map$(TraversableLike.scala:279) ~[scala-library-2.12.15.jar:?]
	at scala.collection.AbstractTraversable.map(Traversable.scala:108) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.deploy.SparkSubmit.downloadResourcesToCurrentDirectory$1(SparkSubmit.scala:427) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$17(SparkSubmit.scala:452) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at scala.Option.map(Option.scala:230) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:452) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1001) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1158) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1167) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ~[spark-core_2.12-3.4.1-amzn-2.jar:3.4.1-amzn-2]
Exception in thread "main" java.io.IOException: Failed to delete: /usr/lib/hudi/./hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar
	at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:150)
	at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:121)
	at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:94)
	at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1232)
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$14(SparkSubmit.scala:436)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at scala.collection.TraversableLike.map(TraversableLike.scala:286)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
	at scala.collection.AbstractTraversable.map(Traversable.scala:108)
	at org.apache.spark.deploy.SparkSubmit.downloadResourcesToCurrentDirectory$1(SparkSubmit.scala:427)
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$17(SparkSubmit.scala:452)
	at scala.Option.map(Option.scala:230)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:452)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1001)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1158)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1167)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

To Reproduce

Steps to reproduce the behavior:

Submit a spark job in EMR on EKS with spark.jars=/usr/lib/hudi/hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar

Expected behavior Spark application to run without any errors

A clear and concise description of what you expected to happen.

Environment Description

• Hudi version : 0.14.0

• Spark version : 3.4.1

• Hive version :

• Hadoop version :

• Storage (HDFS/S3/GCS..) : S3

• Running on Docker? (yes/no) : yes

[ad1happy2go]

@Lakshmi-Holla12 Interesting. Not sure why spark is trying to delete the jar from that location. Can you try putting hudi spark jar in some different location and point that to it.

[ad1happy2go]

@Lakshmi-Holla12 Were you able to resolve this issue?

[Lakshmi-Holla12]

Hi @ad1happy2go, I'm still experiencing the issue with EMR 6.15. By default, Spark jar is located at /usr/lib/hudi, which I am currently using. During the spark-submit, I provide the following parameters --jars /usr/lib/hudi/hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar and --py-files /usr/lib/hudi/<my src code path>. However, I'm not sure why Spark is trying to delete the files. I've tried a few steps to resolve the issue but to no luck:

1. I attempted putting the Hudi Spark jar in a different location and ran into the same error.
2. I removed the --jars parameter during Spark submit, this time it looks for --py-files and throws the same error with not able to delete /usr/lib/hudi/./.
3. I tried setting this configuration spark.hadoop.fs.s3a.deletion.enable, false. This was meant to disable the deletion of temporary files by the S3A filesystem. However, I'm still facing the same issue.
4. I also tried setting spark.hadoop.io.native.lib.available, false. This was to disable the use of native Unix OS commands for deletion. However, I'm still facing the same issue.

Any assistance in resolving this issue would be greatly appreciated.

[ad1happy2go]

@CTTY Any insights on this. thanks in advance.

[Lakshmi-Holla12]

Hi @ad1happy2go @CTTY,

I have been testing EMR on EKS with different EMR versions: emr-6.15.0, emr-6.14.0, emr-6.13.0, and emr-6.11.0.

I found that emr-6.11.0 works fine with Spark 3.3.2 and hudi 0.13.0.

However, emr-6.14.0 and emr-6.13.0, which use Spark 3.4.1 and hudi 0.13.1, started to fail with the same error message: "java.io.IOException: Failed to delete: /usr/lib/hudi/./hudi-spark3-bundle_2.12-0.13.1-amzn-2.jar".

Similarly, emr-6.15.0, which uses Spark 3.4.1 and hudi 0.14.0, also encountered the same issue with the error message: "java.io.IOException: Failed to delete: /usr/lib/hudi/./hudi-spark3-bundle_2.12-0.14.0-amzn-0.jar".

I am not sure what changes in Spark 3.4.1 could be causing the deletion of these files.

[CTTY]

I think we need to understand why Spark is trying to delete Hudi jars here, can you share the previous log so we can see what Spark is doing before this exception?

On the other hand, I wouldn't use /usr/lib/hudi/ to put scripts since the entire directory is owned by root user there and you may run into some weird permission issues