spylon
spylon copied to clipboard
Split "SparkConfiguration" into launcher and runtime configs
It's pretty confusing for the configuration parameters that are required for launching the spark context to be mixed in with the parameters that can be tweaked while the context is active. This is probably an API break so needs to be considered carefully
Maybe we can make the split, but retain a single global (deprecated) object that is the union of the two for a while?
That sounds like the minimal risk approach.
Only a subset of the spark configuration parameters actually do anything if you change them post spark context initialization