add support for Elasticsearch Service on AWS

[codefromthecrypt]

Zipkin 1.12 supports http transport and also hosted thanks to @sethp-jive.

I'm sure folks will soon ask about the dependencies job. Most code in this repo isn't shared because the spark libraries are generally unlike the normal ones.

[codefromthecrypt]

cc @openzipkin/elasticsearch

[codefromthecrypt]

appears http is already supported. we need to verify and look into aws

https://github.com/openzipkin/zipkin-dependencies/blob/master/elasticsearch/src/main/java/zipkin/dependencies/elasticsearch/ElasticsearchDependenciesJob.java#L59

[faiq]

We're running into a problem here where we use Amazon IAM credentials to do communication with Amazon ElasticSearch Service. As of right now, there is no code path to properly load in these credentials from the environment and it is causing our zipkin depenency job to fail.

To fix this we can add something similar to this: https://github.com/openzipkin/zipkin/blob/master/zipkin-autoconfigure/storage-elasticsearch-aws/src/main/java/zipkin/autoconfigure/storage/elasticsearch/aws/ZipkinElasticsearchAwsStorageAutoConfiguration.java

[codefromthecrypt]

@faiq are you up to task to give this a try? Best to use the same environment variables as the server does.

[devinsba]

I looked through a bunch of the issues related to this and came upon a solution.

I am using the following workaround when starting my task in ECS:

[
    {
        "memory": 4096,
        "environment": [
            {"name":"STORAGE_TYPE","value":"elasticsearch"},
            {"name":"ES_HOSTS","value":"http://iam-proxy:9200"},
            {"name":"ES_NODES_WAN_ONLY","value":"true"}
        ],
        "links": [
            "iam-proxy"
        ],
        "essential": true,
        "name": "zipkin-dependencies",
        "image": "quay.io/openzipkin/zipkin-dependencies:1.10",
        "logConfiguration": {
            "logDriver": "awslogs",
            "options": {
                "awslogs-group": "zipkin",
                "awslogs-region": "us-east-1",
                "awslogs-stream-prefix": "zipkin-dependencies"
            }
        },
        "cpu": 2048
    },
    {
        "memory": 1024,
        "portMappings": [
            {
                  "hostPort": 0,
                  "containerPort": 9200,
                  "protocol": "tcp"
            }
        ],
        "command": [
            "./aws-es-proxy",
            "-listen",
            "0.0.0.0:9200",
            "-endpoint",
            "${elasticsearch_url}"
        ],
        "essential": true,
        "name": "iam-proxy",
        "image": "gorillastack/aws-es-proxy",
        "cpu": 512
    }
]

The code for this docker image lives here: https://github.com/abutaha/aws-es-proxy

[jorgheymans]

Wondering if https://github.com/openzipkin/zipkin/pull/3050 would help here ?

[codefromthecrypt]

I guess first step would be getting the thing to read AWS credentials in the first place. I wonder if elasticsearch-hadoop has progresed this at all (or if they would?)

[llinder]

Just bumped my head into this one as a final step in setting up Zipkin with AWS Elasticsearch. Looks like elasticsearch-hadoop hasn't made any progress https://github.com/elastic/elasticsearch-hadoop/issues/626. For now I might either go back to C* or give the signing proxy a try.

[stv-io]

I jumped through a few hoops and resorted to using the proxy to get this to work.

[codefromthecrypt]

It is hard to think elastic/elasticsearch-hadoop#626 is anything but intentionally not addressed. What options do we want to go forward on this? fork? use a java agent to patch the client? switch to something else? In any case it is needless extra work, and also we won't be sure the other driver will be allowed to set headers (ex if we switch to something else)

cc @openzipkin/elasticsearch

[codefromthecrypt]

FWIW I think external (proxy) or patching is the only viable way as ES doesn't seem to progress this topic. Since it is not moving anyway, the code could more easy to patch.

By patch, I mean copy/paste a class that is a part of their driver (noting the LICENSE in the NOTICE file) and edit it such that an interceptor can be added. If someone wants to do this, I'll help review.

[stv-io]

Update from my end:

I have revisited our setup, and currently installing 2.6.2 and pointing directly towards an AWS/ES domain, same one as before. It seems to be working without the intermediate proxy which I have now removed.

AWS/ES versions: Elasticsearch version 7.4 Service software release: R20200910