spylon icon indicating copy to clipboard operation
spylon copied to clipboard

Split "SparkConfiguration" into launcher and runtime configs

Open ericdill opened this issue 8 years ago • 3 comments

It's pretty confusing for the configuration parameters that are required for launching the spark context to be mixed in with the parameters that can be tweaked while the context is active. This is probably an API break so needs to be considered carefully

ericdill avatar Apr 13 '17 13:04 ericdill

Maybe we can make the split, but retain a single global (deprecated) object that is the union of the two for a while?

parente avatar Apr 13 '17 13:04 parente

That sounds like the minimal risk approach.

mariusvniekerk avatar May 24 '17 03:05 mariusvniekerk

Only a subset of the spark configuration parameters actually do anything if you change them post spark context initialization

mariusvniekerk avatar May 24 '17 03:05 mariusvniekerk