docker-zipkin
docker-zipkin copied to clipboard
Error running Zipkin with AWS elasticsearch
Hello guys. I've tried to start zipkin with AWS elasticsearch as a bacend but have such error in UI ERROR: cannot load service names:
Here is my docker run string
docker run -d --name zipkin -p 9411:9411 -e STORAGE_TYPE=elasticsearch -e ES_AWS_DOMAIN="https://search-elk-test-<SOME-ID>.eu-west-1.es.amazonaws.com" openzipkin/zipkin
I deployed AWS elastic as a public domain so each service has access. Please help. Thanks
2017-11-14 12:45:00.281 ERROR 7 --- [nio-9411-exec-4] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.IllegalStateException: ] with root cause
java.lang.IllegalStateException:
at zipkin.autoconfigure.storage.elasticsearch.aws.ElasticsearchDomainEndpoint.get(ElasticsearchDomainEndpoint.java:47) ~[zipkin-autoconfigure-storage-elasticsearch-aws-2.3.0.jar!/:na]
at zipkin2.elasticsearch.ElasticsearchStorage.http(ElasticsearchStorage.java:296) ~[zipkin-storage-elasticsearch-2.3.0.jar!/:na]
at zipkin2.elasticsearch.AutoValue_ElasticsearchStorage.http(AutoValue_ElasticsearchStorage.java:43) ~[zipkin-storage-elasticsearch-2.3.0.jar!/:na]
at zipkin2.elasticsearch.ElasticsearchStorage.ensureIndexTemplates(ElasticsearchStorage.java:283) ~[zipkin-storage-elasticsearch-2.3.0.jar!/:na]
at zipkin2.elasticsearch.AutoValue_ElasticsearchStorage.ensureIndexTemplates(AutoValue_ElasticsearchStorage.java:28) ~[zipkin-storage-elasticsearch-2.3.0.jar!/:na]
at zipkin2.elasticsearch.ElasticsearchStorage.version(ElasticsearchStorage.java:221) ~[zipkin-storage-elasticsearch-2.3.0.jar!/:na]
at zipkin.storage.elasticsearch.http.ElasticsearchHttpStorage.legacyAsyncSpanStore(ElasticsearchHttpStorage.java:149) ~[zipkin-storage-elasticsearch-http-2.3.0.jar!/:na]
at zipkin.internal.V2StorageComponent.spanStore(V2StorageComponent.java:61) ~[io.zipkin.java-zipkin-2.3.0.jar!/:na]
at zipkin.server.ZipkinQueryApiV1.getTraces(ZipkinQueryApiV1.java:108) ~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_144]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_144]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_144]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) ~[spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) ~[spring-webmvc-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) ~[spring-webmvc-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) ~[spring-webmvc-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]```
@devinsba if you have a few mins, can you check this out? otherwise I can try tomorrow
I find a workaround to use ES_HOSTS
It's work for me. But I think that the my problem was connected to AWS credentials . I tried to use instance profile.
I don't use the ES_AWS_DOMAIN
variable but in order to do that, you just give it the name of the ES domain as it gets displayed in the console, not the full URL, then it looks up the URL based on that
Yes I use only domain name but after start he give me 403 Forbidden in UI. Maybe I need some extra permission in my IAM instance profile?
Ok, your example in the issue does not reflect what you just said. In case we are misunderstanding each other.
Instead of:
ES_AWS_DOMAIN="https://search-elk-test-<SOME-ID>.eu-west-1.es.amazonaws.com"
You use:
ES_AWS_DOMAIN="search-elk-test
It will also need describe permission on the ES domain. I can't give you an exact policy though as I use ES_HOSTS
for our configuration
Ok. Sorry for misunderstanding. Basically I start using ES_AWS_DOMAIN="search-elk-test
after I open the issue. Will try to give describe permission on the ES domain. Thanks
I think we are missing a README here: https://github.com/openzipkin/zipkin/tree/master/zipkin-autoconfigure/storage-elasticsearch-aws
which should look similar to here (including a link to perms): https://github.com/openzipkin/zipkin-aws/tree/master/collector-kinesis#collector-kinesis
Also in https://github.com/openzipkin/zipkin/blob/master/zipkin-autoconfigure/storage-elasticsearch-aws/src/main/java/zipkin/autoconfigure/storage/elasticsearch/aws/ElasticsearchDomainEndpoint.java at line 51 code is looking for DomainStatus.Endpoint in json which is not returned by AWS in ES Describe call
I revised the error messages here so that they have more content when something bad happens (Ex at least the response status, which for example 403 could hint at an auth problem)
https://github.com/openzipkin/zipkin/pull/1798