Error running Zipkin with AWS elasticsearch

[Dgadavin]

Hello guys. I've tried to start zipkin with AWS elasticsearch as a bacend but have such error in UI ERROR: cannot load service names: Here is my docker run string docker run -d --name zipkin -p 9411:9411 -e STORAGE_TYPE=elasticsearch -e ES_AWS_DOMAIN="https://search-elk-test-<SOME-ID>.eu-west-1.es.amazonaws.com" openzipkin/zipkin

I deployed AWS elastic as a public domain so each service has access. Please help. Thanks

2017-11-14 12:45:00.281 ERROR 7 --- [nio-9411-exec-4] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.IllegalStateException: ] with root cause

java.lang.IllegalStateException:
	at zipkin.autoconfigure.storage.elasticsearch.aws.ElasticsearchDomainEndpoint.get(ElasticsearchDomainEndpoint.java:47) ~[zipkin-autoconfigure-storage-elasticsearch-aws-2.3.0.jar!/:na]
	at zipkin2.elasticsearch.ElasticsearchStorage.http(ElasticsearchStorage.java:296) ~[zipkin-storage-elasticsearch-2.3.0.jar!/:na]
	at zipkin2.elasticsearch.AutoValue_ElasticsearchStorage.http(AutoValue_ElasticsearchStorage.java:43) ~[zipkin-storage-elasticsearch-2.3.0.jar!/:na]
	at zipkin2.elasticsearch.ElasticsearchStorage.ensureIndexTemplates(ElasticsearchStorage.java:283) ~[zipkin-storage-elasticsearch-2.3.0.jar!/:na]
	at zipkin2.elasticsearch.AutoValue_ElasticsearchStorage.ensureIndexTemplates(AutoValue_ElasticsearchStorage.java:28) ~[zipkin-storage-elasticsearch-2.3.0.jar!/:na]
	at zipkin2.elasticsearch.ElasticsearchStorage.version(ElasticsearchStorage.java:221) ~[zipkin-storage-elasticsearch-2.3.0.jar!/:na]
	at zipkin.storage.elasticsearch.http.ElasticsearchHttpStorage.legacyAsyncSpanStore(ElasticsearchHttpStorage.java:149) ~[zipkin-storage-elasticsearch-http-2.3.0.jar!/:na]
	at zipkin.internal.V2StorageComponent.spanStore(V2StorageComponent.java:61) ~[io.zipkin.java-zipkin-2.3.0.jar!/:na]
	at zipkin.server.ZipkinQueryApiV1.getTraces(ZipkinQueryApiV1.java:108) ~[classes/:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_144]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_144]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_144]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) ~[spring-web-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) ~[spring-webmvc-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) ~[spring-webmvc-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) ~[spring-webmvc-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.3.12.RELEASE.jar!/:4.3.12.RELEASE]```

[codefromthecrypt]

@devinsba if you have a few mins, can you check this out? otherwise I can try tomorrow

[Dgadavin]

I find a workaround to use ES_HOSTS It's work for me. But I think that the my problem was connected to AWS credentials . I tried to use instance profile.

[devinsba]

I don't use the ES_AWS_DOMAIN variable but in order to do that, you just give it the name of the ES domain as it gets displayed in the console, not the full URL, then it looks up the URL based on that

[Dgadavin]

Yes I use only domain name but after start he give me 403 Forbidden in UI. Maybe I need some extra permission in my IAM instance profile?

[devinsba]

Ok, your example in the issue does not reflect what you just said. In case we are misunderstanding each other.

Instead of:

ES_AWS_DOMAIN="https://search-elk-test-<SOME-ID>.eu-west-1.es.amazonaws.com"

You use:

ES_AWS_DOMAIN="search-elk-test

It will also need describe permission on the ES domain. I can't give you an exact policy though as I use ES_HOSTS for our configuration

[Dgadavin]

Ok. Sorry for misunderstanding. Basically I start using ES_AWS_DOMAIN="search-elk-test after I open the issue. Will try to give describe permission on the ES domain. Thanks

[codefromthecrypt]

I think we are missing a README here: https://github.com/openzipkin/zipkin/tree/master/zipkin-autoconfigure/storage-elasticsearch-aws

which should look similar to here (including a link to perms): https://github.com/openzipkin/zipkin-aws/tree/master/collector-kinesis#collector-kinesis

[raevilman]

Also in https://github.com/openzipkin/zipkin/blob/master/zipkin-autoconfigure/storage-elasticsearch-aws/src/main/java/zipkin/autoconfigure/storage/elasticsearch/aws/ElasticsearchDomainEndpoint.java at line 51 code is looking for DomainStatus.Endpoint in json which is not returned by AWS in ES Describe call

[codefromthecrypt]

I revised the error messages here so that they have more content when something bad happens (Ex at least the response status, which for example 403 could hint at an auth problem)

https://github.com/openzipkin/zipkin/pull/1798