DataCleaner
DataCleaner copied to clipboard
Streamline SparkRunner/ApplicationDriver's responsibilities
While doing #1253, we realized that we have quite some overlap between ApplicationDriver
and SparkRunner
. It would be nice if ApplicationDriver
(or maybe smaller, more focused classes?) took care of the individual tasks, while SparkRunner
orchestrated them.
SparkRunner
should probably also do result downloading, which is barely handled currently.
At some point after the 5.0 release it would probably help us a lot if this was improved.