spark-bench icon indicating copy to clipboard operation
spark-bench copied to clipboard

Add a way to benchmark startup/shutdown.

Open cin opened this issue 8 years ago • 2 comments

This will be helpful for developers working on Spark core or connectors.

cin avatar Sep 05 '17 15:09 cin

@cin Would this be an alright API?

spark-bench = {
  spark-submit-config = {
    time-startup = true //default false
    // ....
  }
}

ecurtin avatar Sep 06 '17 14:09 ecurtin

That works.

I was thinking about this a bit more and where I get hung up is when to consider the application as started (i.e. when to stop the timer)? Is it when all the requested executors have been allocated, when the first task gets scheduled, or when the application code starts?

cin avatar Sep 06 '17 14:09 cin