Close batch kills spark job which accumlates files in staging dir

[kraj007]

Hello,

This is particularly not a bug but we are running livy with EMR. when batch completes , it calls the close batch API which kills the batch.

as per default setting, spark.yarn.preserve.staging.files=false This setting will not delete the staging files for jobs that are killed. we saw many files accumulated in staging directory for livy user. We have to remove these files manually.

Do you have any alternate solution for this ?