amazon-neptune-tools icon indicating copy to clipboard operation
amazon-neptune-tools copied to clipboard

add support for sts regional interface endpoints

Open nimble00 opened this issue 2 years ago • 0 comments

Issue #, Not available

Issue Description

  1. Current code does NOT work when it is hosted in a VPC without PublicSubnets (has i-gw) or PrivateSubnets (has nat-gw). It is unable to call AWS STS even when the VPC has InterfaceEndpoints to access AWS services like ECR, STS, S3, DynamoDB.
  2. InterfaceEndpoints allow access ONLY to regional endpoints like - sts.eu-west-1.amazonaws.com etc.
  3. Root cause: the boto3 STS client has a peculiar behavior - even when region_name parameter is specified, it hits the global endpoint - sts.amazon.com instead of regional endpoints. There are two reasons behind this behavior - a. Some of the regions don't have regional STS endpoints itself b. I read somewhere (can't verify correctness!) that it does that for historical reasons - some critical infra was breaking when they tried to move it to regional endpoints.
  4. Solution looks easy - add a PrivateSubnet (NAT-GW)! But what if we don't want that due to security reasons and multitude of other reasons?

Changes / Solution

  1. This issue can be solved by specifying endpoint_url parameter while creating the sts client using boto3. This change has been implemented in this CR.

Testing

  1. The changes are successfully running in Prod in an Enterprise Service. Tested by copying the relevant class (Endpoints) from this codebase to my service package, and implementing the same changes there.
  2. Build is Successful locally -
nimbalr@88665a11506c neptune-python-utils % ./build.sh
...
...
...
  adding: yarl/py.typed (stored 0%)
  adding: yarl/_quoting_c.pyi (deflated 61%)
+ mv neptune_python_utils.zip ../../../../target/neptune_python_utils.zip
+ deactivate
+ unset -f pydoc
+ '[' -z _ ']'
+ PATH=/usr/local/opt/node@14/bin:/Users/nimbalr/.toolbox/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
+ export PATH
+ unset _OLD_VIRTUAL_PATH
+ '[' -z '' ']'
+ hash -r
+ '[' -z _ ']'
+ PS1=
+ export PS1
+ unset _OLD_VIRTUAL_PS1
+ unset VIRTUAL_ENV
+ '[' '!' '' = nondestructive ']'
+ unset -f deactivate
+ popd
~/Documents/codes/amazon-neptune-tools/neptune-python-utils
+ rm -rf temp
nimbalr@88665a11506c neptune-python-utils %

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

nimble00 avatar Jun 29 '22 10:06 nimble00