spark icon indicating copy to clipboard operation
spark copied to clipboard

should call reviveOffer in fine-grained mesos mode after tasks successfully finished

Open xiajunluan opened this issue 12 years ago • 3 comments

[Spark-872]I think we should call reviveOffer in statusUpdate function to request resource, In scheduler.statusUpdate function, it calls reviveOffer only for TASK_LOST and TASK_FAILED, so it need deal with TASK_FINISHED scenario, this improvement will enhance performance significantly.

xiajunluan avatar Aug 13 '13 12:08 xiajunluan

Thank you for your pull request. An admin will review this request soon.

AmplabJenkins avatar Aug 13 '13 13:08 AmplabJenkins

In this mode, Mesos itself should send back offers when our refusals time out after 1 second. Also, we call reviveOffers when a TaskSetManager is submitted. What kind of performance difference have you seen with this, and with what Mesos version?

Calling reviveOffers on every task finish can also be quite expensive because Mesos will list all the free nodes in the cluster for us.

mateiz avatar Aug 13 '13 15:08 mateiz

OK, I'd like to ask the Mesos guys about this, because I think calling reviveOffers on every task-finished will be very expensive on a larger cluster. In older versions of Mesos, the system made a new resource offer as soon as a task finished. Has this changed recently? CC @benh

mateiz avatar Aug 14 '13 18:08 mateiz