hadoop icon indicating copy to clipboard operation
hadoop copied to clipboard

Published 20 hours ago • verified publisher icon mesos

Split MAP and REDUCE tasks into individual mesos tasks

Open tarnfeld opened this issue 10 years ago • 3 comments

This commit splits out the resources for MAP and REDUCE slots into two Mesos tasks instead of one, while still using a single TaskTracker JVM. This allows the idle-slot tracking to operator on MAP and REDUCE slots individually further increasing our ability to release idle resources faster.

This is an implementation of #47.

tarnfeld avatar May 20 '15 23:05 tarnfeld

image

This screenshot was taken while a job was running on a shared cluster, and it's possible to see quite clearly that some Reduce slots (from the 0th task tracker) were revoked and the resources freed, while the maps are still allocated.

tarnfeld avatar May 20 '15 23:05 tarnfeld

Looks pretty good to me. If you've tested it, merge away.

Might be worth bumping the version in the pom too.

brndnmtthws avatar May 21 '15 00:05 brndnmtthws

Thanks for the quick review! I've just rolled this out on one of our clusters so I want to let things settle a bit first, and get some serious traffic through the JT/TTs before saying it's good to go.

tarnfeld avatar May 21 '15 01:05 tarnfeld