storm-deploy
storm-deploy copied to clipboard
nathanmarz
Cannot connect to the Storm UI with Storm 0.9.x
I have encountered many issues attempting to deploy storm to EC2. I have tried several different branches with varying failing results. My last attempt was without specifying a branch (apache-storm-0.9.1-incubating-SNAPSHOT). It appears that the UI daemon is running on the nimbus although it appears that port 8080 is not open for communications. According to the running processes, nimbus appears to be fine but it is difficult to verify without the UI. Additionally, it appears that only one of the two supervisor machines actually is running the daemon.
I noticed 6 "java.lang.ClassNotFoundException: backtype.storm.LocalDRPC" being thrown during the build process, but it really does not seem like this would be the root cause. Please see my commands and config which may help determine whether or not I have made some error.
Lein commands: lein deploy-storm --start --name test lein deploy-storm --attach --name test
.pallet/config.clj: (defpallet :services { :default { :blobstore-provider "aws-s3" :provider "aws-ec2" :environment {:user {:username "storm" ; this must be "storm" :private-key-path "~/.ssh/aws_rsa" :public-key-path "~/.ssh/aws_rsa.pub"} :aws-user-id "xxxx-xxxx-xxxx"} :identity "xxxxxxxxxxxxxxxxxxxxx" :credential "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx" :jclouds.regions "us-west-2" } })
conf/clusters.yaml: nimbus.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu nimbus.hardware: "t1.micro" supervisor.count: 2 supervisor.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu supervisor.hardware: "t1.micro" zookeeper.count: 1 zookeeper.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu zookeeper.hardware: "t1.micro"
conf/storm.yaml storm.local.dir: "/mnt/storm"
Wondering if this is an issue with the port restrictions at the EC2 level. Can you reach port 8080 if you do an wget locally to the UI?
—
Antoni Batchelli,
- twitter: @tbatchelli , @disclojure
--- email: [email protected](mailto:[email protected])
----- web: tbatchelli.org(http://tbatchelli.org/) , disclojure.org(http://disclojure.org/)
----- gpg: public key(http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x482233BD2AFCDF0D)
From: Steve Pisarski [email protected] Reply: nathanmarz/storm-deploy [email protected] Date: January 9, 2014 at 2:57:26 PM To: nathanmarz/storm-deploy [email protected] Subject: [storm-deploy] Cannot connect to the Storm UI (#54)
I have encountered many issues attempting to deploy storm to
EC2. I have tried several different branches with varying failing
results. My last attempt was without specifying a branch (apache-storm-0.9.1-incubating-SNAPSHOT).
It appears that the UI daemon is running on the nimbus although
it appears that port 8080 is not open for communications. According
to the running processes, nimbus appears to be fine but it is difficult
to verify without the UI. Additionally, it appears that only
one of the two supervisor machines actually is running the daemon.I noticed 6 "java.lang.ClassNotFoundException: backtype.storm.LocalDRPC"
being thrown during the build process, but it really does not
seem like this would be the root cause. Please see my commands
and config which may help determine whether or not I have made
some error.Lein commands: lein deploy-storm --start --name test lein deploy-storm --attach --name test
.pallet/config.clj: (defpallet :services { :default { :blobstore-provider "aws-s3" :provider "aws-ec2" :environment {:user {:username "storm" ; this must be "storm"
:private-key-path "~/.ssh/aws_rsa" :public-key-path "~/.ssh/aws_rsa.pub"} :aws-user-id "xxxx-xxxx-xxxx"} :identity "xxxxxxxxxxxxxxxxxxxxx" :credential "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx" :jclouds.regions "us-west-2" } })conf/clusters.yaml: nimbus.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu nimbus.hardware: "t1.micro" supervisor.count: 2 supervisor.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu
supervisor.hardware: "t1.micro" zookeeper.count: 1 zookeeper.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu
zookeeper.hardware: "t1.micro"conf/storm.yaml storm.local.dir: "/mnt/storm"
Reply to this email directly or view it on GitHub: https://github.com/nathanmarz/storm-deploy/issues/54
Thanks for your quick reply!
No luck. I tried with and without the protocol "http://"
storm@ip-172-31-6-67:~$ wget http://localhost:8080 --2014-01-09 23:06:46-- http://localhost:8080/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:8080... failed: Connection refused.
storm@ip-172-31-6-67:~$ wget localhost:8080 --2014-01-09 23:07:20-- http://localhost:8080/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:8080... failed: Connection refused.
Also tried with curl and it returned "couldn't connect to host"
You can reset the ports in the storm.yaml that gets sent.
However if you don't, you can ssh to the nimbus node, and see if storm UI is running, and then check the logs for errors.
On Jan 9, 2014, at 5:03 PM, Antoni Batchelli [email protected] wrote:
Wondering if this is an issue with the port restrictions at the EC2 level. Can you reach port 8080 if you do an wget locally to the UI?
—
Antoni Batchelli,
- twitter: @tbatchelli , @disclojure --- email: [email protected](mailto:[email protected]) ----- web: tbatchelli.org(http://tbatchelli.org/) , disclojure.org(http://disclojure.org/) ----- gpg: public key(http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x482233BD2AFCDF0D)
From: Steve Pisarski [email protected] Reply: nathanmarz/storm-deploy [email protected] Date: January 9, 2014 at 2:57:26 PM To: nathanmarz/storm-deploy [email protected] Subject: [storm-deploy] Cannot connect to the Storm UI (#54)
I have encountered many issues attempting to deploy storm to EC2. I have tried several different branches with varying failing results. My last attempt was without specifying a branch (apache-storm-0.9.1-incubating-SNAPSHOT). It appears that the UI daemon is running on the nimbus although it appears that port 8080 is not open for communications. According to the running processes, nimbus appears to be fine but it is difficult to verify without the UI. Additionally, it appears that only one of the two supervisor machines actually is running the daemon.
I noticed 6 "java.lang.ClassNotFoundException: backtype.storm.LocalDRPC" being thrown during the build process, but it really does not seem like this would be the root cause. Please see my commands and config which may help determine whether or not I have made some error.
Lein commands: lein deploy-storm --start --name test lein deploy-storm --attach --name test
.pallet/config.clj: (defpallet :services { :default { :blobstore-provider "aws-s3" :provider "aws-ec2" :environment {:user {:username "storm" ; this must be "storm" :private-key-path "~/.ssh/aws_rsa" :public-key-path "~/.ssh/aws_rsa.pub"} :aws-user-id "xxxx-xxxx-xxxx"} :identity "xxxxxxxxxxxxxxxxxxxxx" :credential "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx" :jclouds.regions "us-west-2" } })
conf/clusters.yaml: nimbus.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu nimbus.hardware: "t1.micro" supervisor.count: 2 supervisor.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu supervisor.hardware: "t1.micro" zookeeper.count: 1 zookeeper.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu zookeeper.hardware: "t1.micro"
conf/storm.yaml storm.local.dir: "/mnt/storm"
Reply to this email directly or view it on GitHub: https://github.com/nathanmarz/storm-deploy/issues/54 — Reply to this email directly or view it on GitHub.
That is another problem. I cannot find any logs whatsoever. The ~storm/logs/ directory is empty and could not find the files "ui.log" or "nimbus.log". Here is the contents of my built storm.yaml:
storm.local.dir: "/mnt/storm" storm.zookeeper.servers:
- "54.201.32.199" nimbus.host: "54.200.193.86" drpc.servers:
- "172.31.6.67" storm.local.dir: "/mnt/storm"
The /mnt/storm directory is empty. Is this correct? Storm is deployed to "/home/storm/apache-storm-0.9.1-incubating-SNAPSHOT". Could the multiple "storm.local.dir" entries be problematic?
I had lots of problems deploying storm as well similar to what you describe. What eventually worked for me was deploying 0.8.2. The exact same config wouldn't work on the 0.9.0 release candidates, 0.9.0.1 or the latest snapshot. Either the management UI wouldn't load at all - or it would load but show no workers configured etc.
0.8.2 worked perfectly first time.
Ok..
With storm deploy, if you look at ~storm, is that your main directory? Meaning you see the conf/ , bin/ and logs/ directories?
What do you get if you type 'which storm' ?
On Jan 10, 2014, at 9:47 AM, Steve Pisarski [email protected] wrote:
That is another problem. I cannot find any logs whatsoever. The ~storm/logs/ directory is empty and could not find the files "ui.log" or "nimbus.log". Here is the contents of my built storm.yaml:
storm.local.dir: "/mnt/storm" storm.zookeeper.servers:
"54.201.32.199" nimbus.host: "54.200.193.86" drpc.servers: "172.31.6.67" storm.local.dir: "/mnt/storm" The /mnt/storm directory is empty. Is this correct? Storm is deployed to "/home/storm/apache-storm-0.9.1-incubating-SNAPSHOT". Could the multiple "storm.local.dir" entries be problematic?
— Reply to this email directly or view it on GitHub.
Michael, that is indeed the directory structure observed with each version I had attempted; however, 0.8.2 was the only one that contained any log content, which is where I found the Storm UI port number 8772. I had been attempting 8080 in previous attempts of other versions but had no log output that told me that the UI was using a different port (if at all).
Michael, sorry about the 8772, I had forgotten that I changed this myself in the storm.yaml to help with my debugging so ignore my other comments about the port number meaning that I definitely never had luck deploying any storm version other than 0.8.2...
My bet is that the AMI has Java 1.6 installed by default, and storm-0.9.x needs 1.7.
Is this issue fixed ? I seemed to face similar problem while setting up storm 0.9.6 . The only difference I see, is that in my case nimbus logs do get updated . But the access and metrics log remain the same.
