Erik Krogen

I'm a bit confused about why this change is necessary. `yarn.Client` already gathers all Hadoop config files under `HADOOP_CONF_DIR` (and `SPARK_CONF_DIR`) and uploads them, placing them on the classpath of...

Ah, I see. It seems you're using `spark.yarn.populateHadoopClasspath = true`. It looks like it's expected that the Hadoop conf from the node overrides the one from `__hadoop_conf__` in this case:...

Current diff LGTM. We've been testing this out internally (including changes from the umbrella PR as well) and with the recent changes I think this PR looks like it should...

Unfortunately none of I / @thejdeep / @robreeves are committers, just interested parties, so we can't merge. We need a review from a committer as well. @gengliangwang , @cloud-fan ,...

@gengliangwang any comments on the latest diff, after @steven-aerts answered your last question? Seems that this PR is in a very healthy state, I would love to see it merged.

Ping @cloud-fan @gengliangwang @dongjoon-hyun , are any of you available to help review?

Sorry for being late to the party here. The changes to `getAdditionalBlock` look fine to me -- we already check for `WRITE` operation later in the same method, this just...

But the ObserverNN should be aware of the states of DataNodes as well, right? Staleness, maintenance state, etc. It might be slightly stale, but I don't immediately see why that...