storm

storm copied to clipboard

Hi

I would like to access the storm logviewer on one of the supervisors. How do I do that?

(AFAIK the supervisors are started by mesos)

Thanks

It's possible, but it's not wired up. As always, PRs are welcome.

I still haven't managed to get storm to work on mesos, despite days of efforts, so first things first.. :)

IIRC, click logviewer link from nimbus web ui will always try to connect to a constant port (8080? not for sure now), however, mesos launch supervisor in a unique and random generated directory, so the logviewer can't(maybe hack in some way) figure out where to find the correct worker's log.

Certainly that could be fixed.

@yaronr give me about a week to finish up and I'll post how I got it all running ontop of mesos + marathon without hardcoding the nimbus master or anything.

The gist of it is:

1. A marathon application group. One for nimbus, and one for the storm ui. The app for storm ui has a dependency on nimbus, so one starts after the other.
2. A run-storm.py which stores the actual host and port of the nimbus instance in zookeeper using the excellent Kazoo python lib.
3. run-storm.py then writes out the conf/storm.yaml with the proper nimbus.thrift.port an nimbus.host settings before launching nimbus
4. It then writes out ui.port, nimbus.host, and nimbus.thrift.port when launching storm ui.

All of this and I actually don't run it in docker, but natively under marathon, by downloading the built storm distribution and the run-storm.py via the uris flag in marathon json. I'm also chrooting this in a specific zookeeper path and having the framework name include that chroot path, so that multiple storm frameworks can run, potentially as different users, on a single mesos cluster.

In a perfect world, the framework would support a lot of this natively, but I'm a python/golang/c guy, and have never taken a serious stab at java.

SEJeff - Could we use something like mesos-dns to indicate where the nimbus is located? It works well for Myriad and may work well here.

It kind of defeats the point of mesos to require hard coding masters that might change. Mesos DNS is a reasonable workaround, as is my solution of having a python script write to ZK before starting nimbus and another to read from nimbus before starting storm ui.

The framework should do this natively.

Makes sense that the framework should do it natively, I was just suggestion the mesos-dns as a way that the nimbus/ui could be run from any node (given access to the code to run it) and where it landed wouldn't matter to the supervisors as they are pointing to a static name that resolves to a dynamic IP.

True. Thanks for the suggestion

Wouldn't the most straight forward approach be to have the supervisor/executor fire up the logviewer when mesos fires it up?

If no one is working this I would like to get the logview working under mesos.

I've raised an issue with how the PR #38 will handle multiple MesosSupervisors / topologies being scheduled onto the same mesos-slave host, please see: https://github.com/mesos/storm/pull/38#issuecomment-111259423

My proposal for the immediate future is to back out #38 , but I'm not sure what we should do long-term. I have some off-the-cuff thoughts here: https://github.com/mesos/storm/pull/38#issuecomment-112215933

Would appreciate others weighing in with their thoughts.

I've filed a ticket against the Apache Storm project to add support for multiple container-specific logviewers: STORM-1342, thus allowing us to close PR #51.

Will require discussions with the Storm PMC type people to try to come to an acceptable design, I'm not planning on working on that in the next couple of months.