[SPARK-52124] Actively Releasing Disk Space After Application Completio…

[Mrhs121]

…n in Spark Standalone Mode

What changes were proposed in this pull request?

In the spark standalone mode, actively release the disk space occupied by the application in the work directory

Why are the changes needed?

When submitting applications using Spark in standalone mode, a folder is generated under the work directory on each node every time a application is submitted. The naming convention for these folders is, for example, app-20250212191730-0249. These folders contain the resource files that each node downloads from the master node when the application is submitted. Although there is a scheduled cleanup mechanism (spark.worker.cleanup.enabled), it is not immediate. If a large number of application are submitted in a short period of time, and each application depends on a significant amount of external resources, the disk space can be quickly exhausted.

Therefore, I suggest actively deleting the disk space occupied under the work directory after each application is completed.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

No

[Mrhs121]

@cloud-fan Hi~ could you help review this, thanks.

[github-actions[bot]]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!