Error fetching EKS binaries via aws cli from GovCloud

[sl4mmy]

What happened:

We are trying to build custom AMIs for EKS inside an AWS GovCloud account (us-gov-west-1 Region). Unfortunately, due to the way we currently have configured command line access into that account we always end up with AWS_ACCESS_KEY_ID exported in the environment of the make(1) process when running builds in this repo. As a result, the test on line 207 of scripts/install-worker.sh always succeeds and we try to download the EKS binaries via the aws cli instead of wget (see also this comment: https://github.com/awslabs/amazon-eks-ami/pull/175#issuecomment-474510028).

Unfortunately, when downloading the binaries via the aws cli from within GovCloud we get the following error when line 209 of scripts/install-worker.sh is executed: fatal error: An error occurred (400) when calling the HeadObject operation: Bad Request

A larger snippet of the failure is shown below for additional context (I can also post the full build output, if necessary, but this should be easy enough for anyone to reproduce from another GovCloud account):

...
    amazon-ebs: Adding versionlock on: 0:docker-19.03.13ce-1.amzn2
    amazon-ebs: versionlock added: 1
    amazon-ebs: overlay
    amazon-ebs: br_netfilter
    amazon-ebs: net.bridge.bridge-nf-call-ip6tables = 1
    amazon-ebs: net.bridge.bridge-nf-call-iptables = 1
    amazon-ebs: net.ipv4.ip_forward = 1
    amazon-ebs: Downloading binaries from: s3://amazon-eks
    amazon-ebs: AWS cli present - using it to copy binaries from s3.
    amazon-ebs: fatal error: An error occurred (400) when calling the HeadObject operation: Bad Request
==> amazon-ebs: Provisioning step had errors: Running the cleanup provisioner, if present...
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
Build 'amazon-ebs' errored after 5 minutes 9 seconds: Script exited with non-zero exit status: 1.Allowed exit codes are: [0]

==> Wait completed after 5 minutes 9 seconds

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: Script exited with non-zero exit status: 1.Allowed exit codes are: [0]

==> Builds finished but no artifacts were created.
make[1]: *** [k8s] Error 1
make: *** [1.19] Error 2

What you expected to happen:

I would expect that the amazon-eks S3 bucket allows read-only access from GovCloud accounts.

How to reproduce it (as minimally and precisely as possible):

Run the following build in your GovCloud account: make 1.19 aws_region=us-gov-west-1 binary_bucket_region=us-west-2

Anything else we need to know?:

Environment:

• AWS Region: us-gov-west-1
• Instance Type(s): null (we're not overriding it, just using whatever is the default)
• EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): default
• Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): 1.19.13 (build date 2021-09-02)
• AMI Version: default
• Kernel (e.g. uname -a): default
• Release information (run cat /etc/eks/release on a node):

[sl4mmy]

As a work-around we are changing line 148 of eks-worker-al2.json from:

"AWS_ACCESS_KEY_ID={{user `aws_access_key_id`}}",

to:

"AWS_ACCESS_KEY_ID=''",

In order to force scripts/install-worker.sh to download EKS binaries using wget instead of the aws cli.

[i-ate-a-vm]

Bumping this as it's still a problem from anyone in GovCloud. @sl4mmy's workaround works (thank you very much for posting it) but would still be nice to be able to run this regularly in GC.

[rsavage-nozominetworks]

Hitting this issue too, and it's still an issue. Thanks @sl4mmy to confirm your method works, but I agree this is not the way. Would be nice if Gov support was added properly.

[cartermckinnon]

The is a general issue with the credential setup for this packer template -- we need to switch to an instance profile instead of manually replicating local creds from env vars. I'll be AFK for a bit but I'll put up a PR for this in a couple weeks. Sorry for the hassle!

[Sandeepsac]

@cartermckinnon In gov-cloud we facing issue while using instance profile that while connecting with S3 in region specific we getting 403 forbitan EKS version 1.27