spark-bench
spark-bench copied to clipboard
Add a way to benchmark startup/shutdown.
This will be helpful for developers working on Spark core or connectors.
@cin Would this be an alright API?
spark-bench = {
spark-submit-config = {
time-startup = true //default false
// ....
}
}
That works.
I was thinking about this a bit more and where I get hung up is when to consider the application as started (i.e. when to stop the timer)? Is it when all the requested executors have been allocated, when the first task gets scheduled, or when the application code starts?