Edwina Lu
Results
4
comments of
Edwina Lu
Some of the thresholds are meant to be set with heuristic configuration parameters. The stage analysis can be used for multiple heuristics (long task, task skew, execution memory spill, configuration...
The new executor memory metrics have been proposed in SPARK-23206.
@akshayrai is correct. At LinkedIn, we are running Spark 2.3 History Server, which processes Spark applications running Spark 2.1 and 1.6. We have not tried with 1.4.