allow s3 commands to work from non commercial (ie us-gov) environments

[joelddiaz]

Issue #, if available:

N/A

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

when running under something like us-gov-east-1, the latest-binaries.sh script fails b/c it tries to access the us-west-2 bucket from the wrong endpoints.

this can be avoided by setting AWS_ENDPOINT_URL_S3 to point to us-west-2, but you still end up trying to use the gov-cloud creds in the request which would fail with:

$ ./hack/latest-binaries.sh 1.29

An error occurred (InvalidToken) when calling the ListObjectsV2 operation: The provided token is malformed or otherwise invalid.

so, specify to perform an unauthenticated s3 api request b/c the govcloud creds wouldn't work against the commercial cloud endpoints.

in other places in the install-worker.sh script, there are 'aws s3' commands that would fail if running under something like the us-gov-east-1 environment.

similar to the changes to the latest-binaries.sh script, update the 'aws' cli calls to ensure the requests are unsinged (to avoid trying to use us-gov creds against a non-gov endpoint).

Testing Done

Tried using the ./hack/latest-binaries.sh when using govcloud credentials to see things not error out.

Performed a full end-to-end AMI build when running in us-gov-east-1.

[joelddiaz]

@ndbaker1 i went ahead and updated (and tested) the al2023 install-worker.sh script, and some entries in the docs

[joelddiaz]

hopefully the linter gods are pacified this time

[ndbaker1]

/ci build +workflow:os_distros al2,al2023 +workflow:k8s_versions 1.29

[github-actions[bot]]

@ndbaker1 roger that! I've dispatched a workflow. 👍

[github-actions[bot]]

@ndbaker1 the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.29 / al2	success ✅	skipped ⏭️
1.29 / al2023	success ✅	skipped ⏭️

[bradwatsonaws]

I certainly appreciate the GovCloud support! But one thing I don't understand about this process is that the hack/latest-binaries.sh script relies on the AWS CLI for execution while the install-worker.sh scripts all attempt to install the AWS CLI long after the fact. If the AWS CLI isn't installed ahead of time, the process will fail from the start in GovCloud. @cartermckinnon had a very good workaround here to use curl in the event of the AWS CLI command failing.

Is the AWS CLI installation section of the install-worker.sh script really just an attempt to upgrade vs install? If so - ok that makes more sense. From my experience I see that the AL2023 minimal AMIs come with the AWS CLI preinstalled but the AL2 minimal AMIs DO NOT, meaning this approach would not work in GovCloud for AL2. Am I off here?

[joelddiaz]

I certainly appreciate the GovCloud support! But one thing I don't understand about this process is that the hack/latest-binaries.sh script relies on the AWS CLI for execution while the install-worker.sh scripts all attempt to install the AWS CLI long after the fact. If the AWS CLI isn't installed ahead of time, the process will fail from the start in GovCloud. @cartermckinnon had a very good workaround here to use curl in the event of the AWS CLI command failing.

Is the AWS CLI installation section of the install-worker.sh script really just an attempt to upgrade vs install? If so - ok that makes more sense. From my experience I see that the AL2023 minimal AMIs come with the AWS CLI preinstalled but the AL2 minimal AMIs DO NOT, meaning this approach would not work in GovCloud for AL2. Am I off here?

I feel like I'm missing something here regarding the questions...

hack/latest-binaries.sh is a piece that is only used to run locally before kicking of the packer build step(s) for the sole purpose of passing some variables into the packer environment, and the install-worker.sh is the software installation that happens inside the instance/AMI we're building.

I've mostly been working with AL2, and indeed there is no AWS CLI pre-installed on the base-AMI images I've been using.

[bradwatsonaws]

Ah yes, you are correct @joelddiaz. I conflated the usage. My apologies and again appreciate the GovCloud love here!

[jtnord]

would presumably Fix #1536, #1482 and obsolete #1537, #1641

[mebays]

Is this still actively getting worked on. I would support this change. The buckets seem to only be in us-west-2. The aws cli errors off in gov cloud because it won't find a solution for amazon-eks.s3.us-gov-east-1.amazonaws.com, because there is no redirect or bucket in gov cloud. If hitting the endpoint directly with curl to another region amazon-eks.s3.us-east-1.amazonaws.com that ends up being in us-west-2 regardless because it returns a Permanent Redirect to us-west-2.

If you care to look at the debug logs for awscli aws s3api list-objects-v2 --region us-east-1 --bucket amazon-eks --debug will attempt the us-east-1 region get the redirect message grab the Endpoint url (amazon-eks.s3-us-west-2.amazonaws.com) from the message and retry with the retrieved endpoint.

[bradwatsonaws]

Any updates on this PR? hack/latest-binaries.sh has been broken in GovCloud for over 9 months now without using a workaround. @joelddiaz @ndbaker1 @cartermckinnon