Tengfei Huang

cc @Ngone51 , could you please review this PR? Thanks.

cc @Ngone51 , code refactored(add a new `TaskResourceProfile`), could you please take a look? Thanks.

> Can we please add more of a high level design/overview here? > > Also I assume the intent here is to reuse executors since no dynamic allocation - please...

> so I would like to see the issue or this PR description have much more details about design, API, and its behavior. For instance: > > ``` > Does...

Updated the PR's description with more details, please take a look and share your thoughts on this. cc @tgravescs @Ngone51 Proposed new API changes in the description based on previous...

> ``` > /** > * Return resource profile Ids of executors where tasks can be assigned to. > */ > def compatibleExecutorRpIds(rpMgr: ResourceProfileManager): Set[Int] > ``` > > It...

Have been busy for last 2 weeks, get back to this PR. Based on previous discussion, we prefer to remove the `taskOnly` method and create `TaskResourceProfile` implicitly when executor resources...

Gentle ping @tgravescs @Ngone51 . Could you please help to review this PR when you have time? Thanks.

> Mostly looks good. We need to update the docs like: https://github.com/apache/spark/blob/master/docs/configuration.md#stage-level-scheduling-overview It says "the current implementation acquires new executors for each ResourceProfile created" > > Also should update https://github.com/apache/spark/blob/master/docs/spark-standalone.md#stage-level-scheduling-overview...

Thanks @tgravescs and @Ngone51