Matei Zaharia

I think your MESOS_HOME setting code doesn't actually do anything, because the export is only active during the SSH session that you run that in. Did you test it out?...

Hey Tim, This looks good, but I think the export MESOS_HOME stuff is unnecessary. It will only be active during that one SSH command (it does not save the environment...

Good catch; the only thing I'm not 100% sure of is whether this will work when two active jobs depend on the same stage. (This can only happen if two...

Yeah, I think a reference count would be better. We should also make a test where multiple jobs depend on the same stage. For example: ``` val pairs = sc.parallelize(...)...

I think this will have to be done after Spark 0.8, because of the reference-counting issue above. However, @markhamstra has also been looking at the reference counting for some of...

I agree with Stephen here. The addOnCompleteCallback mechanism also makes sure to call your handler if the task throws an exception, which is important. Also, can you add a similar...

Regarding Stephen's comment -- I think it's better to keep PartitionMapper a single object instead of doing functions, in case you need to share state among the setup, map and...

Sorry, just hadn't had a chance to look at it. It looks good but I made two small comments.

Imran, this approach looks good to me, but I'm going to send it to Patrick, who's been looking at monitoring stuff too. I think these are reasonable API calls to...

Hey Jason, I read the design document and this looks very good! Thanks for writing that up. I'm going to go over the code in detail in the next few...