grafana-spark-dashboards icon indicating copy to clipboard operation
grafana-spark-dashboards copied to clipboard

Published 20 hours ago • verified publisher icon hammerlab

Make resource manager pluggable

Open hectcastro opened this issue 9 years ago • 6 comments

Right now, the resource manager referenced in spark.js targets YARN specifically. It would be great to have support for both YARN and Mesos.

hectcastro avatar Mar 05 '15 19:03 hectcastro

Good idea! I've never actually run Spark on Mesos so someone who has will probably be better suited to taking this on.

ryan-williams avatar Mar 05 '15 19:03 ryan-williams

Going to mess with it soon because I want to take advantage of all of the work already done here. Just wanted to put it out there in case anyone else already started down the path.

Thanks, @ryan-williams!

hectcastro avatar Mar 05 '15 19:03 hectcastro

excellent, thanks for your help

ryan-williams avatar Mar 05 '15 19:03 ryan-williams

Hi, we are not using YARN currently, what could be an approach to gather apps ids from master or spark-history server?

seregasheypak avatar Apr 16 '15 14:04 seregasheypak

Good question @seregasheypak. It seems like it should be straightforward to make a request to Spark history server instead of YARN RM, and scrape the results.

Alternatively, once Spark-4435 goes in (which seems like will be soon), maybe there will be a JSON endpoint that you can get a more structured form of that data from?

I'd start with just scraping the history server (unless there's already a JSON endpoint that will work).

Let me know if you are interested in working on that, otherwise I'll see if I can get to it.

ryan-williams avatar Apr 16 '15 14:04 ryan-williams

Got it, will report back in few days.

seregasheypak avatar Apr 16 '15 18:04 seregasheypak