[SPARK-40428][CORE][WIP] Fix shutdown hook in the CoarseGrainedSchedulerBackend

[holdenk]

What changes were proposed in this pull request?

Fix the shutdown hook call through to CoarseGrainedSchedulerBackend

Why are the changes needed?

Sometimes if the driver shuts down abnormally resources may be left dangling.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing tests.

[mridulm]

SparkContext.stop will take care of this - and that is invoked from a shutdown hook already.

SparkContext.stop -> _dagScheduler.stop -> taskScheduler.stop -> backend.stop

[holdenk]

hmm for whatever reason that doesn't seem to be triggering for me, let me take a look through and see where things might be getting lost in that call path @mridulm

[mridulm]

Looks good to me - modulo @dongjoon-hyun's comment. We might want to also wrap FallbackStorage.cleanUp in SparkContext.stop. It is not strictly required for ensuring this specific codepath goes through, but will help ensure stop goes through completely.

[holdenk]

Merged to current dev :)