flintrock
flintrock copied to clipboard
nchammas
Add shortcut to open Spark web UI
I'm on the fence about this, but it might be good to add shortcuts to open common interfaces to Flintrock.
For example, I often find myself opening the Spark web UI. It would be cool if I could do that with some variation of:
./flintrock login cluster --spark-web-ui
Or perhaps a new command is appropriate here:
./flintrock open cluster spark-web-ui
Flintrock would then open a new browser tab and point it at the Spark web UI. We can do this quite easily with click.launch().
Going down this route opens the door to adding other shortcuts which may be useful.
Maybe as a very simple first step approach, we could just print out the URL to the Spark UI. That way people on *nix and OS X can just ctrl-click on the link to open in a browser.
Good idea. One of the planned improvements to Flintrock's describe command is to differentiate the master from the slaves. (Right now you just get a list of nodes.)
With that improvement, users should be able to describe a cluster, copy the master address, tack on :8080, and get the Spark web UI.
It's a good interim solution until we decide on what to do with this issue here.
Why not just print it out when Flintrock exits successfully from the launch command?
Hmm, I would consider that a bit noisy. We are actually planning to reduce the amount of output during a cluster launch in #27, so I wouldn't want to add anything to the output. Also, I think querying information like the master address better belongs under the describe command.
If i remember right, the ec2-script does that and we used it a lot. I'm not talking about adding one per node. I was just thinking something like: You're master is at http://ec2-long-address:8080/
And that's it.
Yup, spark-ec2 does that.
As we reduce the launch output, I'll consider adding a message like this. Perhaps it can replace or become part of the health check that is currently displayed at the end by Flintrock.
Btw @engrean, you can now query the master hostname directly using describe. See the examples in #55, especially the last one.
This is how I'm generating my Spark WebUI links
echo http://$(flintrock describe | egrep "master:(.+)" | grep -Po "\S+$"):8080
Then I can ctrl click on it. I have it in a shell script file though
