[BUG] Allow setting alternate host header (with port) in AWSV4SignerAuth

[daniel-rhoades]

What is the bug?

The fetch_url method within signer.py does not include port information in the signing request - it only specifies uses the host field.

As a result, the signing request is incorrectly generated when connecting to an OpenSearch domain through a proxy or tunnel which uses a different port (e.g. 8443) than the domain is configured for (e.g. 443). This causes an opensearchpy.exceptions.AuthorizationException when attempting to make requests as the expected URL (without a port number) given to the signer doesn't match the request being made (which has the port number).

How can one reproduce the bug?

Using either:

• An OpenSearch domain secured by AWS IAM; or more specifically
• Isolated AWS OpenSearch domain (e.g. deploy into a VPC without an IGW, relying on the appropriate VPC Endpoints instead)

1. Establish a proxy (e.g. a local Nginx proxy) or tunnel (e.g. SSH tunnel to a bastion host) to the domain's endpoint but use a different port from the domain's default port (e.g. use 8443 instead of 443)
2. Ensure the hostname will correctly resolve to the proxy or tunnel, either: a. Add an entry to the local hosts file; or b. Use localhost as the host name below and set verify_certs to False
3. Follow the documented example for connecting to and querying an index, but ensure the client is configured to use the port of the local proxy / tunnel (see example below)

Replace xxx as appropriate for region, host and index. The code assumes appropriate AWS credentials are already available within the environment this code is executed.

from opensearchpy import OpenSearch, RequestsHttpConnection, AWSV4SignerAuth
import boto3

region = 'xxx'
host = f'xxx.{region}.es.amazonaws.com'
port = 8443

credentials = boto3.Session().get_credentials()
auth = AWSV4SignerAuth(credentials, region)

client = OpenSearch(
    hosts=[{'host': host, 'port': port}],
    http_auth=auth,
    use_ssl=True,
    verify_certs=False,
    connection_class=RequestsHttpConnection
)

client.search(
    body=None,
    index='xxx'
)

What is the expected behavior?

Otherwise valid client.search(...) requests should execute without error when connectivity to the OpenSearch domain occurs over a different port (8443) than the default (433) for the given protocol (HTTPS), and requests are being signed by AWSV4SignerAuth.

What is your host/environment?

N/A

Do you have any screenshots?

Exception message:

opensearchpy.exceptions.AuthorizationException: AuthorizationException(403, '{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}')

Do you have any additional context?

If connecting via a proxy or tunnel, ensure it is running on the default port (443), requests then succeed.

Technically, this issue would affect any requests which do not use the protocol's standard port (443). However, in practice this issue is likely to only manifest when connecting to managed AWS OpenSearch domains which will mostly likely be queried without using a proxy / tunnel.

A reasonable solution would be to add the port to the signing request only if it differs from the protocol's default port.

[daniel-rhoades]

Workaround

Run the proxy / tunnel on the same port configured for OpenSearch domain endpoint (e.g. 443 instead of 8443). The downside is that within most environments, this must be run as root / sudo as ports below 1024 are restricted. This may not be possible for some users, regardless, it is bad security practice for the intended purpose.

[dblock]

Would appreciate a PR/fix @daniel-rhoades!

[Xtansia]

I think it's actually the opposite problem. I believe the signer is correctly including the port, because it tries to pull host from the Host header which will include the port, which means it's signing it correctly for the proxy but not correctly for the actual OpenSearch domain.

[saimedhi]

@daniel-rhoades Please feel free to take up this issue and raise a PR with your proposed solution. Thanks :)

[andreaslang]

Created a PR to attempt a fix. Not sure if this is the cleanest way. Interestingly it is OK to keep localhost (does not need to sign for the real host), but the port needs to be changed in the URL used for signing.