The unmeaning Region parameter became mandatory for MRAP endpoint

[gab-david]

Describe the bug

After an upgrade from 1.9.157 to 1.11.4, the parameter Region became mandatory when trying to connect to a MRAP accesspoint.

As described in documentation https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiRegionAccessPointRequests.html

Multi-Region Access Point ARNs don't include an AWS Region

The region should not be mandatory when using MRAP accesspoints.
It should either be ignored or a warning should be raised/logged.

Expected Behavior

The S3 requests are successful without the need to set a region ; the same as it was with SDK 1.9.157.

Current Behavior

Below are the traces that are logged when we create the S3Client() class and create the HeadObject request.

[DEBUG] 2023-04-24 13:49:14.401 ClientConfiguration [140256500507328] Request Compression enabled
[DEBUG] 2023-04-24 13:49:14.401 ClientConfiguration [140256500507328] ClientConfiguration will use MinReqCompression: 10240
[DEBUG] 2023-04-24 13:49:14.401 ClientConfiguration [140256500507328] ClientConfiguration will use SDK Auto Resolved profile: [default] if not specified by users.
[WARN] 2023-04-24 13:49:14.401 ClientConfiguration [140256500507328] Retry Strategy will use the default max attempts.
[INFO] 2023-04-24 13:49:14.401 CurlHandleContainer [140256500507328] Initializing CurlHandleContainer with size 25
[TRACE] 2023-04-24 13:49:14.403 Aws::Endpoint::DefaultEndpointProvider [140256378881600] Endpoint bool eval parameter: UseFIPS = 0
[TRACE] 2023-04-24 13:49:14.403 Aws::Endpoint::DefaultEndpointProvider [140256378881600] Endpoint bool eval parameter: UseDualStack = 0
[TRACE] 2023-04-24 13:49:14.403 Aws::Endpoint::DefaultEndpointProvider [140256378881600] Endpoint bool eval parameter: UseArnRegion = 0
[TRACE] 2023-04-24 13:49:14.403 Aws::Endpoint::DefaultEndpointProvider [140256378881600] Endpoint bool eval parameter: DisableMultiRegionAccessPoints = 0
[TRACE] 2023-04-24 13:49:14.403 Aws::Endpoint::DefaultEndpointProvider [140256378881600] Endpoint str eval parameter: Bucket = arn:aws:s3::123456789012:accesspoint/mfzwi23gnjvgw.mrap
[ERROR] 2023-04-24 13:49:14.403 HeadObject [140256378881600] A region must be set when sending requests to S3.

Workaround with Region=invalid-region:

[DEBUG] 2023-04-24 14:16:37.120 ClientConfiguration [140069699732160] Request Compression enabled
[DEBUG] 2023-04-24 14:16:37.120 ClientConfiguration [140069699732160] ClientConfiguration will use MinReqCompression: 10240
[DEBUG] 2023-04-24 14:16:37.120 ClientConfiguration [140069699732160] ClientConfiguration will use SDK Auto Resolved profile: [default] if not specified by users.
[WARN] 2023-04-24 14:16:37.120 ClientConfiguration [140069699732160] Retry Strategy will use the default max attempts.
[INFO] 2023-04-24 14:16:37.120 CurlHandleContainer [140069699732160] Initializing CurlHandleContainer with size 25
[TRACE] 2023-04-24 14:16:37.121 Aws::Endpoint::DefaultEndpointProvider [140069578106432] Endpoint str eval parameter: Region = invalid-region
[TRACE] 2023-04-24 14:16:37.121 Aws::Endpoint::DefaultEndpointProvider [140069578106432] Endpoint bool eval parameter: UseFIPS = 0
[TRACE] 2023-04-24 14:16:37.121 Aws::Endpoint::DefaultEndpointProvider [140069578106432] Endpoint bool eval parameter: UseDualStack = 0
[TRACE] 2023-04-24 14:16:37.121 Aws::Endpoint::DefaultEndpointProvider [140069578106432] Endpoint bool eval parameter: UseArnRegion = 0
[TRACE] 2023-04-24 14:16:37.121 Aws::Endpoint::DefaultEndpointProvider [140069578106432] Endpoint bool eval parameter: DisableMultiRegionAccessPoints = 0
[TRACE] 2023-04-24 14:16:37.121 Aws::Endpoint::DefaultEndpointProvider [140069578106432] Endpoint str eval parameter: Bucket = arn:aws:s3::123456789012:accesspoint/mfzwi23gnjvgw.mrap
[DEBUG] 2023-04-24 14:16:37.121 Aws::Endpoint::DefaultEndpointProvider [140069578106432] Endpoint rules engine evaluated the endpoint: https://mfzwi23gnjvgw.mrap.accesspoint.s3-global.amazonaws.com
[TRACE] 2023-04-24 14:16:37.121 Aws::Endpoint::DefaultEndpointProvider [140069578106432] Endpoint rules evaluated props: {"authSchemes":[{"disableDoubleEncoding":true,"name":"sigv4a","signingName":"s3","signingRegionSet":["*"]}]}
[TRACE] 2023-04-24 14:16:37.121 AWSClient [140069578106432] No content body, content-length headers
[DEBUG] 2023-04-24 14:16:37.121 AWSAuthSymmetricV4Signer [140069578106432] Note: Http payloads are not being signed. signPayloads=0 http scheme=https
[DEBUG] 2023-04-24 14:16:37.122 AWSClient [140069578106432] Request Successfully signed
[TRACE] 2023-04-24 14:16:37.122 CurlHttpClient [140069578106432] Making request to https://mfzwi23gnjvgw.mrap.accesspoint.s3-global.amazonaws.com/**
...

Reproduction Steps

In aws-sdk-cpp/generated/tests/s3-gen-tests/S3EndpointProviderTests.cpp, remove EpParam("Region", "us-east-1") for any mrap test case.

Possible Solution

Rework the file aws-sdk-cpp/tools/code-generation/endpoints/s3-2006-03-01.endpoint-rule-set.json o that MRAP arn are correctly handled.

Additional Information/Context

Workaround is to set any (even invalid) text value in Region parameter.

AWS CPP SDK version used

1.11.64

Compiler and Version used

gcc 11.3

Operating System and version

Linux 5.15

[SergeyRyabinin]

Hi @gab-david ,

I'm really sorry you are facing this new limitation. Thank you a lot for diving deep into the implementation details of how it works now. The current AWS SDKs approach is to consider that the Region config parameter is always present. And we (as the SDK), don't own the endpoint rules file. We have a workaround for legacy configs of overriden endpoint, where we set a fake region, however for the MRAP use case, it is hard to guess if user is going to make MRAP calls at the client construction time.

Long story short, your workaround is valid and I would suggest to keep it. I will check internally if SDKs could relax the relatively new limitation of a required region, even for MRAP, but I'm afraid it won't be changed.

Best regards, Sergey

[gab-david]

Hi @SergeyRyabinin,

Thanks for your quick answer.

I also see that the region is used to set the partition in rules file and then aws_map_region_to_partition() is called. Is (or will be) MRAP supported for these partitions ? Like in China or for aws-us-gov even if they don't appear in the list of 'Only the following AWS Regions are supported' https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiRegionAccessPointRestrictions.html

For this issue, we'll use the workaround with the same fake region.

Best regards, Gabriel

[SergeyRyabinin]

Hi @gab-david ,

As you pointed out, region is used to determine the partition. However, our current API does not provide an explicit way of selecting a partition.

When/if the SDK is going to provide such support, we will have to introduce an additional API parameter or figure out the correct partition from the ARN, request object. Sorry, I cannot comment / provide guidance on this topic at this moment.

Best regards, Sergey