Mridul M

Have not looked in detail, but there are a bunch of other places where `externalAccums` is directly used from - are they also susceptible to these issues ? If yes,...

I agree, given the usecase, `CopyOnWriteArrayList` might be a good option `LuciferYang`. +CC @JoshRosen - in case you have thoughts on this.

Thanks for the detailed analysis @JoshRosen - I agree with your analysis. I saw two cases where this could be happening - * test code or user code calling spark-internal...

+CC @zoelin7, @xkrogen FYI

`SparkContext.stop` will take care of this - and that is invoked from a shutdown hook already. SparkContext.stop -> _dagScheduler.stop -> taskScheduler.stop -> backend.stop

Looks good to me - modulo @dongjoon-hyun's comment. We might want to also wrap `FallbackStorage.cleanUp` in `SparkContext.stop`. It is not strictly required for ensuring this specific codepath goes through, but...

I am not sure I follow what the code snippet is trying to do. Changing the code to : ``` private def receiveLoop() { Executors.newSingleThreadExecutor(threadFactory).execute(new MessageLoop) } ``` does cause...

A few points: * Plugins are initialized as part of `Executor` construction, but after uncaught exception handler is set. * If anything other than `NonFatal` is thrown in `receive` (like...

I will try to reproduce this locally - is there anything else specific I need to know to do so ? Or is the instructions in SPARK-40320 sufficient ? Thx

I agree @Ngone51, it should have :-) I am trying to reproduce this locally and see if we are missing something - or there is some nuance behind the observed...