AWS Amazon OpenSearch Service endpoint format

[ecc256]

trafficstars

Guys, Need a little help here. What would be the right endpoint format for helm deployment with AWS Amazon OpenSearch Service (successor to Amazon Elasticsearch Service)? The latter is in VPC, allows connect from local subnets only, w/o auth and allows http. It's helm3 if it matters.

elasticsearch:
  enabled: false

search:
  # Proxy information needed for the search
  proxy:
    # search.proxy.endpoint -- Endpoint of the search proxy (i.e., ES endpoint etc.)
    # You should only need to change this, if you don't use the version in this chart.
    endpoint: "http://<name>.us-east-1.es.amazonaws.com"
    endpoint: "http://<name>.us-east-1.es.amazonaws.com:80"
    endpoint: "<name>.us-east-1.es.amazonaws.com:80"

[boring-cyborg[bot]]

Thanks for opening your first issue here!

[feng-tao]

maybe worth pinging the slack and see if anyone tries on amazon opensearch? But in general I think opensearch is ES 7.13 successor meaning someone has already upgraded amundsen seaarch to ES 7.x

[ecc256]

maybe worth pinging the slack and see if anyone tries on amazon opensearch?

We are AWS ELK 7.10, didn't have a courage to upgrade AWS opensearch yet...

But in general I think opensearch is ES 7.13 successor meaning someone has already upgraded amundsen search to ES 7.x

Could you tell the slack channel name, please? I'll ask there too.

And a variant of the original question: What is the endpoint format for a standalone standard ELK 7.X listening on non-standard port?

[NiravLangaliya]

it is on port 443.

    host = os.environ.get('PROXY_ENDPOINT')
    port = 443
    use_ssl = True
    verify_certs = True
    region = os.environ.get('AWS_REGION')
    credentials = boto3.Session().get_credentials()

https://github.com/amundsen-io/amundsen/pull/1141

https://github.com/amundsen-io/amundsen/blob/996837933b59d96ee4a47549c21f6147556967f0/search/search_service/config.py#L63

[ecc256]

I've looked up config.py noticed few things, like:

    PROXY_USER = os.environ.get('CREDENTIALS_PROXY_USER', 'elastic')
    PROXY_PASSWORD = os.environ.get('CREDENTIALS_PROXY_PASSWORD', 'elastic')

and

    To assess AWS Elasticsearch domain correctly you need to setup AWS credentials with
    a role that enables reading and writting to Elasticsearch Service domain;
    see the sample CloudFormation IAM policy below::
        {
            "Version": "2012-10-17",
            "Statement": [
                {
                "Effect": "Allow",
                "Principal": {
                    "AWS": [
                    "arn:aws:iam::123456789012:user/test-user"
                    ]
                },
                "Action": [
                    "es:ESHttpGet",
                    "es:ESHttpPut"
                ],
                "Resource": "arn:aws:es:us-west-1:987654321098:domain/test-domain/test-index/_search"
                }
            ]
        }

We do run AWS Elasticsearch Service in VPC. It does accept connections from VPC subnets and peered VPC subnets only. with IAM policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:423854963575:domain/amundsen/*"
    }
  ]
}

It's useless to ask if it's going to work or not, I assume.

Is there a way to ssh into search pod and check if proxy works properly?

[NiravLangaliya]

We have this policy set and we are able to access ES client. ES Version : Elasticsearch 7.10

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:<XXXXXXX>:domain/<XXXXXXX>/*"
    }
  ]
}

[ecc256]

@NiravLangaliya I.e. no any changes on client side (AWS ELK login/auth related), right? Good to know, it works for you in any case. In mine, even ELK indexes were not created...