Tom Arnfeld

Interesting thread. Roles is one way to go, but as you say @hansbogert it's basically static partitioning at the cluster level. We actually have the described issue not only across...

> it will still hoard resources as chances are still pretty high that both mappers and reducers are busy in all but the larger clusters. This is indeed the case,...

Launching a TaskTracker per task would cause things to slow to a snails pace, the turnaround time for an individual task (well, depends on your workload) is so low compared...

This project evolved from the original experiments I believe, and was taken out of the Mesos code base a couple of years ago. I'm not sure of the details of...

@DarinJ I'd be very much interested in that. We've got loads of individual job benchmarking data that we could plug into a system like that!

> @tarnfeld what are you trying to guard here? It looks like your worried something could be added be the tracker between the idleCounter >= idleCheckMax and the scheduler.killTracker (maybe...

Sure if you could share your thoughts even just here in a command that'd be great.

I have a feeling that this issue may now be resolved on master, could you report back @hermansc? I think the commit that introduced this was removed.

 This screenshot was taken while a job was running on a shared cluster, and it's possible to see quite clearly that some Reduce slots (from the 0th task tracker)...

Thanks for the quick review! I've just rolled this out on one of our clusters so I want to let things settle a bit first, and get some serious traffic...