storm-deploy
storm-deploy copied to clipboard
nathanmarz
failed with code 500
I have tried many different setups with older JDK 6, creating my own PEM or using the AWS PEM and creating a .pub. For some reason when I try to deploy I get this,
INFO execute - Output: /home/stormdb/.ssh/pscinternal.pem.pub
DEBUG execute - out => "/home/stormdb/.ssh/pscinternal.pem.pub\n" INFO execute - Output: /home/stormdb/.ssh/pscinternal.pem
DEBUG execute - out => "/home/stormdb/.ssh/pscinternal.pem\n" INFO execute - Output: storm
DEBUG execute - out => "storm\n" INFO execute - Output: /home/stormdb/.ssh/pscinternal.pem.pub
DEBUG execute - out => "/home/stormdb/.ssh/pscinternal.pem.pub\n" INFO execute - Output: /home/stormdb/.ssh/pscinternal.pem
DEBUG execute - out => "/home/stormdb/.ssh/pscinternal.pem\n" INFO execute - Output: /home/stormdb/.ssh/pscinternal.pem.pub
DEBUG execute - out => "/home/stormdb/.ssh/pscinternal.pem.pub\n" INFO execute - Output: /home/stormdb/.ssh/pscinternal.pem
DEBUG execute - out => "/home/stormdb/.ssh/pscinternal.pem\n" DEBUG jclouds - Found jclouds sshj driver DEBUG jclouds - extensions (:log4j :slf4j :sshj) DEBUG jclouds - options [:jclouds.regions "us-east-1" :blobstore-provider "aws-s3"] ERROR logging - Exception in thread "main" ERROR logging - com.google.inject.CreationException: Guice creation errors:
- org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot be used as a key; It is not fully specified.
1 error (form-init1617305676035910732.clj:1) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5440) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5415) ERROR logging - at clojure.lang.Compiler.load(Compiler.java:5857) ERROR logging - at clojure.lang.Compiler.loadFile(Compiler.java:5820) ERROR logging - at clojure.main$load_script.invoke(main.clj:221) ERROR logging - at clojure.main$init_opt.invoke(main.clj:226) ERROR logging - at clojure.main$initialize.invoke(main.clj:254) ERROR logging - at clojure.main$null_opt.invoke(main.clj:279) ERROR logging - at clojure.main$main.doInvoke(main.clj:354) ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:422) ERROR logging - at clojure.lang.Var.invoke(Var.java:369) ERROR logging - at clojure.lang.AFn.applyToHelper(AFn.java:165) ERROR logging - at clojure.lang.Var.applyTo(Var.java:482) ERROR logging - at clojure.main.main(main.java:37) ERROR logging - Caused by: com.google.inject.CreationException: Guice creation errors:
- org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot be used as a key; It is not fully specified.
1 error ERROR logging - at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435) ERROR logging - at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154) ERROR logging - at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106) ERROR logging - at com.google.inject.Guice.createInjector(Guice.java:95) ERROR logging - at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:324) ERROR logging - at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:262) ERROR logging - at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:524) ERROR logging - at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:504) ERROR logging - at org.jclouds.compute2$compute_service.doInvoke(compute2.clj:92) ERROR logging - at clojure.lang.RestFn.applyTo(RestFn.java:147) ERROR logging - at clojure.core$apply.doInvoke(core.clj:548) ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:562) ERROR logging - at pallet.compute.jclouds$eval5952$fn__5954.invoke(jclouds.clj:720) ERROR logging - at clojure.lang.MultiFn.invoke(MultiFn.java:167) ERROR logging - at pallet.compute$compute_service.doInvoke(compute.clj:36) ERROR logging - at clojure.lang.RestFn.applyTo(RestFn.java:140) ERROR logging - at clojure.core$apply.invoke(core.clj:542) ERROR logging - at pallet.configure$compute_service_from_map.invoke(configure.clj:176) ERROR logging - at clojure.lang.Var.invoke(Var.java:365) ERROR logging - at clojure.lang.AFn.applyToHelper(AFn.java:163) ERROR logging - at clojure.lang.Var.applyTo(Var.java:482) ERROR logging - at clojure.core$apply.invoke(core.clj:540) ERROR logging - at pallet.compute$compute_service_from_map.doInvoke(compute.clj:185) ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:409) ERROR logging - at backtype.storm.provision$mk_aws.invoke(provision.clj:120) ERROR logging - at backtype.storm.provision$_main.doInvoke(provision.clj:123) ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:483) ERROR logging - at clojure.lang.Var.invoke(Var.java:381) ERROR logging - at user$eval5$fn__7.invoke(form-init1617305676035910732.clj:1) ERROR logging - at user$eval5.invoke(form-init1617305676035910732.clj:1) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5424) ERROR logging - ... 13 more
Anyone else experiencing this issue?
I don’t think storm-deploy works can authorize .pem keys (the one you specify in the ~/.pallet/config.clj file). Have you tried creating a .pub key from your .pem key and using this one instead?
openssl rsa -in privkey.pem -pubout > key.pub
Storm deploy creates and authorizes a separate user, and uses your key to authorize it an authenticate later on. There is no need to use your amazon .pem key, but if you want to use it, you need to turn it into a .pub.
Toni.
On March 21, 2014 at 6:05:58 AM, troyrob ([email protected]) wrote:
I have tried many different setups with older JDK 6, creating my own PEM or using the AWS
PEM and creating a .pub. For some reason when I try to deploy I get this,INFO execute - Output: /home/stormdb/.ssh/pscinternal.pem.pub
DEBUG execute - out => "/home/stormdb/.ssh/pscinternal.pem.pub\n" INFO execute - Output: /home/stormdb/.ssh/pscinternal.pem
DEBUG execute - out => "/home/stormdb/.ssh/pscinternal.pem\n" INFO execute - Output: storm
DEBUG execute - out => "storm\n" INFO execute - Output: /home/stormdb/.ssh/pscinternal.pem.pub
DEBUG execute - out => "/home/stormdb/.ssh/pscinternal.pem.pub\n" INFO execute - Output: /home/stormdb/.ssh/pscinternal.pem
DEBUG execute - out => "/home/stormdb/.ssh/pscinternal.pem\n" INFO execute - Output: /home/stormdb/.ssh/pscinternal.pem.pub
DEBUG execute - out => "/home/stormdb/.ssh/pscinternal.pem.pub\n" INFO execute - Output: /home/stormdb/.ssh/pscinternal.pem
DEBUG execute - out => "/home/stormdb/.ssh/pscinternal.pem\n" DEBUG jclouds - Found jclouds sshj driver DEBUG jclouds - extensions (:log4j :slf4j :sshj) DEBUG jclouds - options [:jclouds.regions "us-east-1" :blobstore-provider "aws-s3"]
ERROR logging - Exception in thread "main" ERROR logging - com.google.inject.CreationException: Guice creation errors:
- org.jclouds.rest.RestContext cannot
be used as a key; It is not fully specified.1 error (form-init1617305676035910732.clj:1) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5440) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5415) ERROR logging - at clojure.lang.Compiler.load(Compiler.java:5857) ERROR logging - at clojure.lang.Compiler.loadFile(Compiler.java:5820) ERROR logging - at clojure.main$load_script.invoke(main.clj:221) ERROR logging - at clojure.main$init_opt.invoke(main.clj:226) ERROR logging - at clojure.main$initialize.invoke(main.clj:254) ERROR logging - at clojure.main$null_opt.invoke(main.clj:279) ERROR logging - at clojure.main$main.doInvoke(main.clj:354) ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:422) ERROR logging - at clojure.lang.Var.invoke(Var.java:369) ERROR logging - at clojure.lang.AFn.applyToHelper(AFn.java:165) ERROR logging - at clojure.lang.Var.applyTo(Var.java:482) ERROR logging - at clojure.main.main(main.java:37) ERROR logging - Caused by: com.google.inject.CreationException: Guice creation
errors:
- org.jclouds.rest.RestContext cannot
be used as a key; It is not fully specified.1 error ERROR logging - at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435)
ERROR logging - at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154)
ERROR logging - at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106)
ERROR logging - at com.google.inject.Guice.createInjector(Guice.java:95) ERROR logging - at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:324)
ERROR logging - at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:262)
ERROR logging - at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:524)
ERROR logging - at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:504)
ERROR logging - at org.jclouds.compute2$compute_service.doInvoke(compute2.clj:92)
ERROR logging - at clojure.lang.RestFn.applyTo(RestFn.java:147) ERROR logging - at clojure.core$apply.doInvoke(core.clj:548) ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:562) ERROR logging - at pallet.compute.jclouds$eval5952$fn__5954.invoke(jclouds.clj:720)
ERROR logging - at clojure.lang.MultiFn.invoke(MultiFn.java:167) ERROR logging - at pallet.compute$compute_service.doInvoke(compute.clj:36) ERROR logging - at clojure.lang.RestFn.applyTo(RestFn.java:140) ERROR logging - at clojure.core$apply.invoke(core.clj:542) ERROR logging - at pallet.configure$compute_service_from_map.invoke(configure.clj:176)
ERROR logging - at clojure.lang.Var.invoke(Var.java:365) ERROR logging - at clojure.lang.AFn.applyToHelper(AFn.java:163) ERROR logging - at clojure.lang.Var.applyTo(Var.java:482) ERROR logging - at clojure.core$apply.invoke(core.clj:540) ERROR logging - at pallet.compute$compute_service_from_map.doInvoke(compute.clj:185)
ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:409) ERROR logging - at backtype.storm.provision$mk_aws.invoke(provision.clj:120)
ERROR logging - at backtype.storm.provision$_main.doInvoke(provision.clj:123)
ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:483) ERROR logging - at clojure.lang.Var.invoke(Var.java:381) ERROR logging - at user$eval5$fn__7.invoke(form-init1617305676035910732.clj:1)
ERROR logging - at user$eval5.invoke(form-init1617305676035910732.clj:1) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5424) ERROR logging - ... 13 moreAnyone else experiencing this issue?
Reply to this email directly or view it on GitHub: https://github.com/nathanmarz/storm-deploy/issues/59
@troyrob I believe this error is related to JRE 1.7.0_u51 breaking something in Guice. Try installing JRE 1.7.0_u45 and see if the problem persists. I just hit this bug.
So I created a new stormdeploy VM and only installed JRE 1.7.0_u45. I now get a new error.
ERROR logging - Exception in thread "main" ERROR logging - org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 500, error: AWSError{requestId='3d28ceeb-383f-4480-aafb-aeb8e9f56fe4', requestToken='null', code='InternalError', message='An internal error has occurred', context='{Response=, Errors=}'} (form-init6307871250580972215.clj:1) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5440) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5415) ERROR logging - at clojure.lang.Compiler.load(Compiler.java:5857) ERROR logging - at clojure.lang.Compiler.loadFile(Compiler.java:5820) ERROR logging - at clojure.main$load_script.invoke(main.clj:221) ERROR logging - at clojure.main$init_opt.invoke(main.clj:226) ERROR logging - at clojure.main$initialize.invoke(main.clj:254) ERROR logging - at clojure.main$null_opt.invoke(main.clj:279) ERROR logging - at clojure.main$main.doInvoke(main.clj:354) ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:422) ERROR logging - at clojure.lang.Var.invoke(Var.java:369) ERROR logging - at clojure.lang.AFn.applyToHelper(AFn.java:165) ERROR logging - at clojure.lang.Var.applyTo(Var.java:482) ERROR logging - at clojure.main.main(main.java:37) ERROR logging - Caused by: org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 500, error: AWSError{requestId='3d28ceeb-383f-4480-aafb-aeb8e9f56fe4', requestToken='null', code='InternalError', message='An internal error has occurred', context='{Response=, Errors=}'} ERROR logging - at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:78) ERROR logging - at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:71) ERROR logging - at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:197) ERROR logging - at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:167) ERROR logging - at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:135) ERROR logging - at java.util.concurrent.FutureTask.run(Unknown Source) ERROR logging - at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ERROR logging - at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ERROR logging - at java.lang.Thread.run(Unknown Source) ERROR logging - at org.jclouds.concurrent.config.DescribingExecutorService.submit(DescribingExecutorService.java:89) ERROR logging - at org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:132) ERROR logging - at org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54) ERROR logging - at org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73) ERROR logging - at org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:255) ERROR logging - at org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:149) ERROR logging - at com.sun.proxy.$Proxy75.authorizeSecurityGroupIngressInRegion(Unknown Source) ERROR logging - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ERROR logging - at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ERROR logging - at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ERROR logging - at java.lang.reflect.Method.invoke(Unknown Source) ERROR logging - at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:170) ERROR logging - at com.sun.proxy.$Proxy76.authorizeSecurityGroupIngressInRegion(Unknown Source) ERROR logging - at backtype.storm.security$authorize_group.invoke(security.clj:116) ERROR logging - at backtype.storm.security$authorize_group.invoke(security.clj:111) ERROR logging - at backtype.storm.provision$start_with_nodes_BANG_.invoke(provision.clj:77) ERROR logging - at backtype.storm.provision$start_BANG_.invoke(provision.clj:91) ERROR logging - at backtype.storm.provision$main$fn__8422.invoke(provision.clj:144) ERROR logging - at backtype.storm.provision$main.doInvoke(provision.clj:130) ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:437) ERROR logging - at clojure.lang.Var.invoke(Var.java:373) ERROR logging - at user$eval5$fn__7.invoke(form-init6307871250580972215.clj:1) ERROR logging - at user$eval5.invoke(form-init6307871250580972215.clj:1) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5424) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5415) ERROR logging - at clojure.lang.Compiler.load(Compiler.java:5857) ERROR logging - at clojure.lang.Compiler.loadFile(Compiler.java:5820) ERROR logging - at clojure.main$load_script.invoke(main.clj:221) ERROR logging - at clojure.main$init_opt.invoke(main.clj:226) ERROR logging - at clojure.main$initialize.invoke(main.clj:254) ERROR logging - at clojure.main$null_opt.invoke(main.clj:279) ERROR logging - at clojure.main$main.doInvoke(main.clj:354) ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:422) ERROR logging - at clojure.lang.Var.invoke(Var.java:369) ERROR logging - at clojure.lang.AFn.applyToHelper(AFn.java:165) ERROR logging - at clojure.lang.Var.applyTo(Var.java:482) ERROR logging - at clojure.main.main(main.java:37) ERROR logging - at org.jclouds.concurrent.config.DescribingExecutorService.submit(DescribingExecutorService.java:89) ERROR logging - at org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:132) ERROR logging - at org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54) ERROR logging - at org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73) ERROR logging - at org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:255) ERROR logging - at org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:149) ERROR logging - at com.sun.proxy.$Proxy75.authorizeSecurityGroupIngressInRegion(Unknown Source) ERROR logging - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ERROR logging - at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ERROR logging - at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ERROR logging - at java.lang.reflect.Method.invoke(Unknown Source) ERROR logging - at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:170) ERROR logging - at com.sun.proxy.$Proxy76.authorizeSecurityGroupIngressInRegion(Unknown Source) ERROR logging - at backtype.storm.security$authorize_group.invoke(security.clj:116) ERROR logging - at backtype.storm.security$authorize_group.invoke(security.clj:111) ERROR logging - at backtype.storm.provision$start_with_nodes_BANG.invoke(provision.clj:77) ERROR logging - at backtype.storm.provision$start_BANG.invoke(provision.clj:91) ERROR logging - at backtype.storm.provision$_main$fn__8422.invoke(provision.clj:144) ERROR logging - at backtype.storm.provision$_main.doInvoke(provision.clj:130) ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:437) ERROR logging - at clojure.lang.Var.invoke(Var.java:373) ERROR logging - at user$eval5$fn__7.invoke(form-init6307871250580972215.clj:1) ERROR logging - at user$eval5.invoke(form-init6307871250580972215.clj:1) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5424) ERROR logging - ... 13 more
created a branch 'update-clojure-jclouds' with an updated version of jclouds that doesn't have the issue with java 1.7.0_u51. Could you give it a try and see if the error goes away?
Using the .rsa key should have worked with 1.7.5_u45 though, as I tested with that version. A 500 error seems an error on AWS's side.
So I have created a new VM with for storm deploy with latest version of java and updated clojure clouds but still get the errors. They are creating key pairs so it is able to authorize into EC2.
I'd need more info to track down what this could be. There are two authentication steps that are needed. One is to connect to AWS, the other is to connect to the nodes created in AWS. When pallet cannot authenticate on the nodes after creation, it destroys them as they will not be further configurable.
Can you verify if the nodes are getting created and then destroyed? Initially you were using a PEM keypair, are you still using that or moved to RSA?
Moved over to RSA. I think I might of found the issue. I do have availability sets in my EC2 environment. Could this be an issue and if so is there a way to configure to utilize the availability set?
Had the same problem - JRE 1.7.0_55 and _51 and OpenJDK 1.7.0_51 resulted in org.jclouds.rest.RestContext cannot be used as a key; It is not fully specified.
Switched to Oracle JRE 1.7.0_45 and had success. Creating a VM environment just for Storm management.
I'm having this exact same problem even when using oracle JRE 1.7.0_45. I'm trying to set this up on an AWS VM, so far I can't get storm-deploy working.
Is there any fix for this issue? I tried with JDK 1.8.0_31. Still getting the same issue.
